2024-03-09T12:35:04.426 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-09T12:35:04.427 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2024-03-08_16:18:36-rados-wip-yuri8-testing-2024-03-06-1329-distro-default-smithi/7587021 --owner scheduled_yuriw@teuthology --description rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} --name yuriw-2024-03-08_16:18:36-rados-wip-yuri8-testing-2024-03-06-1329-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-03-08_16:18:36-rados-wip-yuri8-testing-2024-03-06-1329-distro-default-smithi/7587021/orig.config.yaml 2024-03-09T12:35:04.449 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-09T12:35:04.551 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-08_16:18:36-rados-wip-yuri8-testing-2024-03-06-1329-distro-default-smithi/7587021 branch: wip-yuri8-testing-2024-03-06-1329 description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} email: yweinste@redhat.com first_in_suite: false job_id: '7587021' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-03-08_16:18:36-rados-wip-yuri8-testing-2024-03-06-1329-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri8-testing-2024-03-06-1329 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 6b01fe95f3aa686f1951a0c5828907a52930b92a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 6b01fe95f3aa686f1951a0c5828907a52930b92a workunit: branch: wip-yuri8-testing-2024-03-06-1329 sha1: 6b01fe95f3aa686f1951a0c5828907a52930b92a owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1270 sha1: 6b01fe95f3aa686f1951a0c5828907a52930b92a sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri8-testing-2024-03-06-1329 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_6b01fe95f3aa686f1951a0c5828907a52930b92a/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 6b01fe95f3aa686f1951a0c5828907a52930b92a targets: smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEN5uJTU89OO65vN6U9GqKJMNEG4MsVHa5bZQ7w2+Rhh6g4fVNeoAfzK25WmUu4UiDLKUku6wEe+amWomUPmEyA= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-08_16:18:36 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-09T12:35:04.551 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_6b01fe95f3aa686f1951a0c5828907a52930b92a/qa; will attempt to use it 2024-03-09T12:35:04.551 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_6b01fe95f3aa686f1951a0c5828907a52930b92a/qa/tasks 2024-03-09T12:35:04.552 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-09T12:35:04.553 INFO:teuthology.task.internal:Checking packages... 2024-03-09T12:35:04.576 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '6b01fe95f3aa686f1951a0c5828907a52930b92a' 2024-03-09T12:35:04.576 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-09T12:35:04.576 INFO:teuthology.packaging:ref: None 2024-03-09T12:35:04.576 INFO:teuthology.packaging:tag: None 2024-03-09T12:35:04.576 INFO:teuthology.packaging:branch: wip-yuri8-testing-2024-03-06-1329 2024-03-09T12:35:04.576 INFO:teuthology.packaging:sha1: 6b01fe95f3aa686f1951a0c5828907a52930b92a 2024-03-09T12:35:04.576 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri8-testing-2024-03-06-1329 2024-03-09T12:35:04.723 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1920.g6b01fe95 2024-03-09T12:35:04.725 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-09T12:35:04.731 INFO:teuthology.task.internal:no buildpackages task found 2024-03-09T12:35:04.731 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-09T12:35:04.773 INFO:teuthology.task.internal:Saving configuration 2024-03-09T12:35:04.783 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-09T12:35:04.799 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T12:35:04.824 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:18:36-rados-wip-yuri8-testing-2024-03-06-1329-distro-default-smithi/7587021', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-09 12:29:47.993507', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEN5uJTU89OO65vN6U9GqKJMNEG4MsVHa5bZQ7w2+Rhh6g4fVNeoAfzK25WmUu4UiDLKUku6wEe+amWomUPmEyA='} 2024-03-09T12:35:04.825 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-09T12:35:04.830 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-03-09T12:35:04.830 INFO:teuthology.run_tasks:Running task console_log... 2024-03-09T12:35:04.882 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8d436c8f70>, signals=[15]) 2024-03-09T12:35:04.882 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-09T12:35:04.888 INFO:teuthology.task.internal:Opening connections... 2024-03-09T12:35:04.889 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-03-09T12:35:04.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:35:04.979 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-09T12:35:04.986 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-03-09T12:35:05.046 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-03-09T12:35:05.046 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-03-09T12:35:05.102 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-03-09T12:35:05.102 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2024-03-09T12:35:05.102 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-03-09T12:35:05.102 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-03-09T12:35:05.102 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2024-03-09T12:35:05.102 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2024-03-09T12:35:05.102 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-09T12:35:05.102 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-03-09T12:35:05.102 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2024-03-09T12:35:05.102 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-09T12:35:05.102 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-03-09T12:35:05.103 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-09T12:35:05.103 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-09T12:35:05.103 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-09T12:35:05.103 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-03-09T12:35:05.138 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-09T12:35:05.147 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-09T12:35:05.206 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-09T12:35:05.207 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-03-09T12:35:05.222 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-09T12:35:05.240 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-09T12:35:05.240 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-03-09T12:35:05.350 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-09T12:35:05.381 INFO:teuthology.run_tasks:Running task kernel... 2024-03-09T12:35:05.417 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-09T12:35:05.417 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-09T12:35:05.417 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-09T12:35:05.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T12:35:05.434 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-03-09T12:35:05.491 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-361.el9.x86_64 2024-03-09T12:35:05.491 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.14.0-361.el9.x86_64 2024-03-09T12:35:05.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2024-03-09T12:35:07.676 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-09T12:35:09.276 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-09T12:35:10.309 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-09T12:35:12.084 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-09T12:35:16.026 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.1 MB 00:00 2024-03-09T12:35:17.594 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 15 kB 00:00 2024-03-09T12:35:18.662 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-09T12:35:23.603 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-09T12:35:25.809 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-09T12:35:25.809 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-09T12:35:25.912 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:35:25.924 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:35:25.924 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-03-09T12:35:25.924 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:35:25.924 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:35:25.924 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-09T12:35:25.924 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-09T12:35:25.924 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-09T12:35:25.924 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-09T12:35:25.924 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-09T12:35:25.924 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:35:25.925 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:35:25.925 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:35:25.925 INFO:teuthology.orchestra.run.smithi162.stdout:Install 4 Packages 2024-03-09T12:35:25.925 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:35:25.925 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 100 M 2024-03-09T12:35:25.925 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 123 M 2024-03-09T12:35:25.925 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:35:27.579 INFO:teuthology.orchestra.run.smithi162.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 4.0 MB/s | 6.0 MB 00:01 2024-03-09T12:35:49.234 INFO:teuthology.orchestra.run.smithi162.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 921 kB/s | 21 MB 00:23 2024-03-09T12:36:05.822 INFO:teuthology.orchestra.run.smithi162.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 895 kB/s | 33 MB 00:38 2024-03-09T12:36:10.696 INFO:teuthology.orchestra.run.smithi162.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 902 kB/s | 39 MB 00:44 2024-03-09T12:36:10.696 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:36:10.696 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.2 MB/s | 100 MB 00:44 2024-03-09T12:36:11.577 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:36:11.956 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:36:11.956 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:36:13.620 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:36:13.621 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:36:16.541 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:36:17.197 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-09T12:36:17.320 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-09T12:36:20.082 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-09T12:36:20.262 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-09T12:36:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-09T12:36:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-09T12:36:29.957 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-09T12:36:59.425 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-09T12:36:59.425 INFO:teuthology.orchestra.run.smithi162.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-09T12:36:59.425 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:36:59.509 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-09T12:36:59.905 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-09T12:36:59.905 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-09T12:36:59.905 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-09T12:36:59.905 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-09T12:37:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-09T12:37:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:37:00.351 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:37:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-09T12:37:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-09T12:37:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-09T12:37:00.351 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-09T12:37:00.352 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:37:00.352 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:37:00.909 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-03-09T12:37:01.068 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-09T12:37:01.068 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-09T12:37:01.068 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-09T12:37:01.068 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-09T12:37:01.068 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-09T12:37:01.069 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-09T12:37:01.069 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi162.front.sepia.ceph.com, path=None, version=distro) 2024-03-09T12:37:01.069 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-03-09T12:37:01.190 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-09T12:37:01.190 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-09T12:37:01.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo rpm -qi grub2-tools 2024-03-09T12:37:01.225 INFO:teuthology.orchestra.run.smithi162.stdout:Name : grub2-tools 2024-03-09T12:37:01.225 INFO:teuthology.orchestra.run.smithi162.stdout:Epoch : 1 2024-03-09T12:37:01.225 INFO:teuthology.orchestra.run.smithi162.stdout:Version : 2.06 2024-03-09T12:37:01.225 INFO:teuthology.orchestra.run.smithi162.stdout:Release : 68.el9 2024-03-09T12:37:01.225 INFO:teuthology.orchestra.run.smithi162.stdout:Architecture: x86_64 2024-03-09T12:37:01.225 INFO:teuthology.orchestra.run.smithi162.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-09T12:37:01.225 INFO:teuthology.orchestra.run.smithi162.stdout:Group : Unspecified 2024-03-09T12:37:01.225 INFO:teuthology.orchestra.run.smithi162.stdout:Size : 8264254 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:License : GPLv3+ 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:Packager : builder@centos.org 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:Vendor : CentOS 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:Summary : Support tools for GRUB. 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:Description : 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-09T12:37:01.226 INFO:teuthology.orchestra.run.smithi162.stdout:hardware devices. 2024-03-09T12:37:01.227 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:37:01.227 INFO:teuthology.orchestra.run.smithi162.stdout:This subpackage provides tools for support of all platforms. 2024-03-09T12:37:01.227 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-09T12:37:01.228 INFO:teuthology.task.kernel:Updating grub on smithi162 to boot 5.14.0-427.el9.x86_64 2024-03-09T12:37:01.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-09T12:37:01.817 INFO:teuthology.orchestra.run.smithi162.stderr:Generating grub configuration file ... 2024-03-09T12:37:03.196 INFO:teuthology.orchestra.run.smithi162.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-09T12:37:03.227 INFO:teuthology.orchestra.run.smithi162.stderr:done 2024-03-09T12:37:03.229 DEBUG:teuthology.orchestra.run.smithi162:> mktemp 2024-03-09T12:37:03.244 INFO:teuthology.orchestra.run.smithi162.stdout:/tmp/tmp.MBzf6rU6mb 2024-03-09T12:37:03.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MBzf6rU6mb 2024-03-09T12:37:03.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0666 /tmp/tmp.MBzf6rU6mb 2024-03-09T12:37:03.455 DEBUG:teuthology.orchestra.remote:smithi162:/tmp/tmp.MBzf6rU6mb is 6KB 2024-03-09T12:37:03.504 DEBUG:teuthology.orchestra.run.smithi162:> rm -fr /tmp/tmp.MBzf6rU6mb 2024-03-09T12:37:03.518 DEBUG:teuthology.orchestra.run.smithi162:> sudo /bin/ls /boot/loader/entries || true 2024-03-09T12:37:03.583 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-09T12:37:03.583 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-09T12:37:03.583 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-09T12:37:03.583 INFO:teuthology.orchestra.run.smithi162.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-09T12:37:03.584 DEBUG:teuthology.orchestra.run.smithi162:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-09T12:37:03.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo shutdown -r now 2024-03-09T12:37:33.746 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-09T12:37:33.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-09T12:37:33.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:37:52.287 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-09T12:38:01.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-09T12:38:01.297 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:38:04.354 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-09T12:38:16.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-09T12:38:16.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:38:34.783 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.162 2024-03-09T12:38:49.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-09T12:38:49.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:38:53.090 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-03-09T12:38:53.582 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-09T12:38:53.583 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-09T12:38:53.583 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-03-09T12:38:53.597 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-427.el9.x86_64 2024-03-09T12:38:53.597 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-09T12:38:53.597 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-09T12:38:53.597 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-09T12:38:54.598 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-09T12:38:54.598 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-09T12:38:54.692 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-03-09T12:38:54.734 DEBUG:teuthology.parallel:result is None 2024-03-09T12:38:54.734 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-09T12:38:54.742 INFO:teuthology.task.internal:Creating test directory... 2024-03-09T12:38:54.742 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-09T12:38:54.758 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-09T12:38:54.840 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-09T12:38:54.874 INFO:teuthology.task.internal:Creating archive directory... 2024-03-09T12:38:54.874 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-09T12:38:54.899 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-09T12:38:54.909 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-09T12:38:54.910 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-09T12:38:54.967 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:38:54.980 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:38:54.982 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-09T12:38:54.989 INFO:teuthology.task.internal:Configuring sudo... 2024-03-09T12:38:54.989 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-09T12:38:55.049 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-09T12:38:55.059 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-09T12:38:55.059 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-09T12:38:55.104 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-09T12:38:55.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-09T12:38:55.237 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-09T12:38:55.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-09T12:38:55.365 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-09T12:38:55.365 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-09T12:38:55.432 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-03-09T12:38:55.529 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-09T12:38:55.861 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-09T12:38:55.868 INFO:teuthology.task.internal:Starting timer... 2024-03-09T12:38:55.869 INFO:teuthology.run_tasks:Running task pcp... 2024-03-09T12:38:55.902 INFO:teuthology.run_tasks:Running task selinux... 2024-03-09T12:38:55.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-03-09T12:38:55.987 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-03-09T12:38:55.989 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-09T12:38:55.989 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-03-09T12:38:56.004 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-03-09T12:38:56.004 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi162.front.sepia.ceph.com': 'permissive'} 2024-03-09T12:38:56.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-09T12:38:56.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T12:38:56.072 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-09T12:38:56.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-03-09T12:38:56.165 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-09T12:38:56.174 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-09T12:38:56.179 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-09T12:38:56.291 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-09T12:38:56.306 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-09T12:38:56.307 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi162.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-09T12:43:05.678 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi162.front.sepia.ceph.com')] 2024-03-09T12:43:05.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-09T12:43:05.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-09T12:43:05.769 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-03-09T12:43:05.845 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-03-09T12:43:05.845 INFO:teuthology.run_tasks:Running task clock... 2024-03-09T12:43:05.856 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-09T12:43:05.856 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-09T12:43:05.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-09T12:43:05.916 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-09T12:43:05.933 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-09T12:43:05.962 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-03-09T12:43:05.976 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-03-09T12:43:05.993 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-09T12:43:06.010 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-09T12:43:06.067 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-03-09T12:43:06.071 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-09T12:43:06.071 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-03-09T12:43:06.071 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-09T12:43:06.071 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-09T12:43:06.071 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-09T12:43:06.072 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-09T12:43:06.072 INFO:teuthology.run_tasks:Running task install... 2024-03-09T12:43:06.080 DEBUG:teuthology.task.install:project ceph 2024-03-09T12:43:06.081 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '6b01fe95f3aa686f1951a0c5828907a52930b92a'}} 2024-03-09T12:43:06.081 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '6b01fe95f3aa686f1951a0c5828907a52930b92a'} 2024-03-09T12:43:06.081 INFO:teuthology.task.install:Using flavor: default 2024-03-09T12:43:06.087 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-09T12:43:06.087 INFO:teuthology.task.install:extra packages: [] 2024-03-09T12:43:06.087 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6b01fe95f3aa686f1951a0c5828907a52930b92a', 'tag': None, 'wait_for_package': False} 2024-03-09T12:43:06.087 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6b01fe95f3aa686f1951a0c5828907a52930b92a 2024-03-09T12:43:06.229 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-03-06-1329/6b01fe95f3aa686f1951a0c5828907a52930b92a/centos/9/flavors/default/ 2024-03-09T12:43:06.229 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1920.g6b01fe95 2024-03-09T12:43:06.411 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-03-06-1329/6b01fe95f3aa686f1951a0c5828907a52930b92a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-03-06-1329/6b01fe95f3aa686f1951a0c5828907a52930b92a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri8-testing-2024-03-06-1329/6b01fe95f3aa686f1951a0c5828907a52930b92a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-09T12:43:06.411 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-09T12:43:06.411 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-09T12:43:06.440 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-09T12:43:06.440 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6b01fe95f3aa686f1951a0c5828907a52930b92a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-09T12:43:06.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-09T12:43:06.589 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-09T12:43:06.655 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-03-09T12:43:06.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-03-09T12:43:06.895 INFO:teuthology.orchestra.run.smithi162.stdout:74 files removed 2024-03-09T12:43:06.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw 2024-03-09T12:43:07.710 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 176 kB/s | 82 kB 00:00 2024-03-09T12:43:08.077 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-09T12:43:08.519 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-03-09T12:43:09.071 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-09T12:43:10.472 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-09T12:43:11.971 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 6.9 MB/s | 8.0 MB 00:01 2024-03-09T12:43:13.938 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-09T12:43:18.172 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.1 MB 00:00 2024-03-09T12:43:19.523 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 15 kB 00:00 2024-03-09T12:43:21.545 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-09T12:43:26.476 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-09T12:43:29.020 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:43:29.023 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:43:29.024 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-09T12:43:29.024 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:43:29.024 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:43:29.024 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 9.8 M 2024-03-09T12:43:29.024 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-03-09T12:43:29.024 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 3.4 M 2024-03-09T12:43:29.025 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 3.1 M 2024-03-09T12:43:29.025 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-09T12:43:29.025 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options 2024-03-09T12:43:29.025 INFO:teuthology.orchestra.run.smithi162.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-09T12:43:29.025 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 5.4 M 2024-03-09T12:43:29.025 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 20 M 2024-03-09T12:43:29.025 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 25 k 2024-03-09T12:43:29.026 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-09T12:43:29.026 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-09T12:43:29.026 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-09T12:43:29.026 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 745 k 2024-03-09T12:43:29.026 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-09T12:43:29.027 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-09T12:43:29.027 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-09T12:43:29.027 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 501 k 2024-03-09T12:43:29.027 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-09T12:43:29.027 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 5.1 M 2024-03-09T12:43:29.027 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-09T12:43:29.028 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-09T12:43:29.028 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-09T12:43:29.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse 2024-03-09T12:43:29.028 INFO:teuthology.orchestra.run.smithi162.stdout: x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 45 k 2024-03-09T12:43:29.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 131 k 2024-03-09T12:43:29.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 162 k 2024-03-09T12:43:29.028 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-09T12:43:29.029 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 320 k 2024-03-09T12:43:29.029 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 298 k 2024-03-09T12:43:29.029 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 99 k 2024-03-09T12:43:29.029 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-09T12:43:29.029 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-09T12:43:29.029 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-09T12:43:29.030 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-09T12:43:29.030 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-03-09T12:43:29.030 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-09T12:43:29.030 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:43:29.030 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:43:29.030 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:43:29.030 INFO:teuthology.orchestra.run.smithi162.stdout:Install 30 Packages 2024-03-09T12:43:29.030 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 2 Packages 2024-03-09T12:43:29.031 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:43:29.031 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 58 M 2024-03-09T12:43:29.031 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:43:29.726 INFO:teuthology.orchestra.run.smithi162.stdout:(1/32): ceph-base-19.0.0-1920.g6b01fe95.el9.x86 9.9 MB/s | 5.4 MB 00:00 2024-03-09T12:43:29.760 INFO:teuthology.orchestra.run.smithi162.stdout:(2/32): ceph-selinux-19.0.0-1920.g6b01fe95.el9. 759 kB/s | 25 kB 00:00 2024-03-09T12:43:29.835 INFO:teuthology.orchestra.run.smithi162.stdout:(3/32): libcephfs2-19.0.0-1920.g6b01fe95.el9.x8 9.7 MB/s | 745 kB 00:00 2024-03-09T12:43:29.886 INFO:teuthology.orchestra.run.smithi162.stdout:(4/32): libradosstriper1-19.0.0-1920.g6b01fe95. 9.8 MB/s | 501 kB 00:00 2024-03-09T12:43:30.269 INFO:teuthology.orchestra.run.smithi162.stdout:(5/32): librgw2-19.0.0-1920.g6b01fe95.el9.x86_6 13 MB/s | 5.1 MB 00:00 2024-03-09T12:43:30.311 INFO:teuthology.orchestra.run.smithi162.stdout:(6/32): python3-ceph-argparse-19.0.0-1920.g6b01 1.1 MB/s | 45 kB 00:00 2024-03-09T12:43:30.361 INFO:teuthology.orchestra.run.smithi162.stdout:(7/32): python3-ceph-common-19.0.0-1920.g6b01fe 2.6 MB/s | 131 kB 00:00 2024-03-09T12:43:30.420 INFO:teuthology.orchestra.run.smithi162.stdout:(8/32): python3-cephfs-19.0.0-1920.g6b01fe95.el 2.7 MB/s | 162 kB 00:00 2024-03-09T12:43:30.479 INFO:teuthology.orchestra.run.smithi162.stdout:(9/32): python3-rados-19.0.0-1920.g6b01fe95.el9 5.4 MB/s | 320 kB 00:00 2024-03-09T12:43:30.545 INFO:teuthology.orchestra.run.smithi162.stdout:(10/32): python3-rbd-19.0.0-1920.g6b01fe95.el9. 4.4 MB/s | 298 kB 00:00 2024-03-09T12:43:30.596 INFO:teuthology.orchestra.run.smithi162.stdout:(11/32): python3-rgw-19.0.0-1920.g6b01fe95.el9. 1.9 MB/s | 99 kB 00:00 2024-03-09T12:43:30.704 INFO:teuthology.orchestra.run.smithi162.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 979 kB/s | 106 kB 00:00 2024-03-09T12:43:30.746 INFO:teuthology.orchestra.run.smithi162.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-09T12:43:30.780 INFO:teuthology.orchestra.run.smithi162.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-09T12:43:30.839 INFO:teuthology.orchestra.run.smithi162.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-09T12:43:30.880 INFO:teuthology.orchestra.run.smithi162.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-09T12:43:30.914 INFO:teuthology.orchestra.run.smithi162.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-09T12:43:31.039 INFO:teuthology.orchestra.run.smithi162.stdout:(18/32): ceph-radosgw-19.0.0-1920.g6b01fe95.el9 5.3 MB/s | 9.8 MB 00:01 2024-03-09T12:43:31.548 INFO:teuthology.orchestra.run.smithi162.stdout:(19/32): ceph-common-19.0.0-1920.g6b01fe95.el9. 8.4 MB/s | 20 MB 00:02 2024-03-09T12:43:31.824 INFO:teuthology.orchestra.run.smithi162.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 44 kB/s | 35 kB 00:00 2024-03-09T12:43:32.024 INFO:teuthology.orchestra.run.smithi162.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 195 kB/s | 216 kB 00:01 2024-03-09T12:43:32.241 INFO:teuthology.orchestra.run.smithi162.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 741 kB/s | 309 kB 00:00 2024-03-09T12:43:32.266 INFO:teuthology.orchestra.run.smithi162.stdout:(23/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 62 kB/s | 44 kB 00:00 2024-03-09T12:43:32.308 INFO:teuthology.orchestra.run.smithi162.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-03-09T12:43:32.391 INFO:teuthology.orchestra.run.smithi162.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 809 kB/s | 67 kB 00:00 2024-03-09T12:43:32.558 INFO:teuthology.orchestra.run.smithi162.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-03-09T12:43:32.583 INFO:teuthology.orchestra.run.smithi162.stdout:(27/32): liboath-2.6.7-2.el9.x86_64.rpm 153 kB/s | 49 kB 00:00 2024-03-09T12:43:32.634 INFO:teuthology.orchestra.run.smithi162.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-09T12:43:32.717 INFO:teuthology.orchestra.run.smithi162.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.4 MB/s | 4.4 MB 00:00 2024-03-09T12:43:32.968 INFO:teuthology.orchestra.run.smithi162.stdout:(30/32): librados2-19.0.0-1920.g6b01fe95.el9.x8 10 MB/s | 3.4 MB 00:00 2024-03-09T12:43:33.010 INFO:teuthology.orchestra.run.smithi162.stdout:(31/32): thrift-0.15.0-2.el9.x86_64.rpm 3.7 MB/s | 1.6 MB 00:00 2024-03-09T12:43:33.135 INFO:teuthology.orchestra.run.smithi162.stdout:(32/32): librbd1-19.0.0-1920.g6b01fe95.el9.x86_ 7.5 MB/s | 3.1 MB 00:00 2024-03-09T12:43:33.142 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:43:33.142 INFO:teuthology.orchestra.run.smithi162.stdout:Total 14 MB/s | 58 MB 00:04 2024-03-09T12:43:33.176 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:43:33.214 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:43:33.214 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:43:33.691 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:43:33.691 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:43:35.166 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:43:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-09T12:43:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-09T12:43:35.315 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-09T12:43:35.337 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-09T12:43:35.354 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-09T12:43:35.533 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-09T12:43:35.593 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-09T12:43:35.965 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-09T12:43:36.016 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:19.0.0-1920.g6b01fe95.el9.x86_64 7/34 2024-03-09T12:43:36.091 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:19.0.0-1920.g6b01fe95.el9.x86_64 7/34 2024-03-09T12:43:36.179 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:19.0.0-1920.g6b01fe95.el9.x86_64 8/34 2024-03-09T12:43:36.210 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:19.0.0-1920.g6b01fe95.el9.x86_64 9/34 2024-03-09T12:43:36.264 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:19.0.0-1920.g6b01fe95.el9.x86_64 9/34 2024-03-09T12:43:36.293 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-09T12:43:36.337 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:19.0.0-1920.g6b01fe95.el9. 11/34 2024-03-09T12:43:36.407 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:19.0.0-1920.g6b01fe95.el9.x86_64 12/34 2024-03-09T12:43:36.429 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:19.0.0-1920.g6b01fe95.el9.x86_6 13/34 2024-03-09T12:43:36.499 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1920.g6b01fe95.el9.x86_6 13/34 2024-03-09T12:43:36.527 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-09T12:43:36.639 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-09T12:43:36.670 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-09T12:43:36.711 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-09T12:43:36.743 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-09T12:43:36.772 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-09T12:43:36.805 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-09T12:43:37.131 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-09T12:43:37.160 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:19.0.0-1920.g6b01fe95.el9.x86_64 22/34 2024-03-09T12:43:37.249 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:19.0.0-1920.g6b01fe95.el9.x86_64 22/34 2024-03-09T12:43:37.281 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:19.0.0-1920.g6b01fe95.el9.x86_64 23/34 2024-03-09T12:43:37.323 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-09T12:43:37.786 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-09T12:43:37.812 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:19.0.0-1920.g6b01fe95.el9.x86_64 26/34 2024-03-09T12:43:37.867 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:19.0.0-1920.g6b01fe95.el9.x86_64 26/34 2024-03-09T12:43:37.944 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:19.0.0-1920.g6b01fe95.el9.x86_64 27/34 2024-03-09T12:43:37.973 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:19.0.0-1920.g6b01fe95.el9.x8 28/34 2024-03-09T12:43:40.135 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-1920.g6b01fe95.el9.x86_64 29/34 2024-03-09T12:43:40.177 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:19.0.0-1920.g6b01fe95.el9.x86_64 29/34 2024-03-09T12:43:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.0.0-1920.g6b01fe95.el9.x86_64 29/34 2024-03-09T12:43:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:19.0.0-1920.g6b01fe95.el9.x86_64 30/34 2024-03-09T12:43:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:19.0.0-1920.g6b01fe95.el9.x86_64 30/34 2024-03-09T12:43:40.881 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-09T12:43:40.881 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-09T12:43:40.881 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:43:40.924 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:19.0.0-1920.g6b01fe95.el9.x86_64 31/34 2024-03-09T12:43:50.754 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1920.g6b01fe95.el9.x86_64 31/34 2024-03-09T12:43:50.754 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-03-09T12:43:50.755 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-03-09T12:43:50.755 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-03-09T12:43:50.755 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-03-09T12:43:50.755 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-03-09T12:43:50.755 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2024-03-09T12:43:50.755 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-03-09T12:43:50.755 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:43:51.680 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:19.0.0-1920.g6b01fe95.el9.x86_64 32/34 2024-03-09T12:43:51.705 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1920.g6b01fe95.el9.x86_64 32/34 2024-03-09T12:43:51.705 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T12:43:51.705 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-09T12:43:51.705 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-09T12:43:51.705 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-09T12:43:51.705 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:43:51.707 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-09T12:43:51.791 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-09T12:43:51.792 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-09T12:43:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-09T12:43:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/34 2024-03-09T12:43:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:19.0.0-1920.g6b01fe95.el9.x86_64 2/34 2024-03-09T12:43:52.992 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:19.0.0-1920.g6b01fe95.el9.x86_64 3/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:19.0.0-1920.g6b01fe95.el9.x86_64 4/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:19.0.0-1920.g6b01fe95.el9.x86_64 5/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:19.0.0-1920.g6b01fe95.el9.x86_6 6/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:19.0.0-1920.g6b01fe95.el9.x86_64 7/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1920.g6b01fe95.el9. 8/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:19.0.0-1920.g6b01fe95.el9.x8 9/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:19.0.0-1920.g6b01fe95.el9.x86_64 10/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:19.0.0-1920.g6b01fe95.el9.x86_64 11/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:19.0.0-1920.g6b01fe95.el9.x86_64 12/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:19.0.0-1920.g6b01fe95.el9.x86_64 13/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-09T12:43:52.993 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-09T12:43:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-09T12:43:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-09T12:43:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-09T12:43:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-09T12:43:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-09T12:43:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-09T12:43:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-09T12:43:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-09T12:43:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-09T12:43:52.994 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-09T12:43:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-09T12:43:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-09T12:43:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:19.0.0-1920.g6b01fe95.el9.x86_64 31/34 2024-03-09T12:43:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-09T12:43:52.995 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:19.0.0-1920.g6b01fe95.el9.x86_64 33/34 2024-03-09T12:43:54.175 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-09T12:43:54.175 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:43:54.175 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-03-09T12:43:54.176 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.176 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.176 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:43:54.176 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-09T12:43:54.176 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.176 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.176 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.176 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.176 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-09T12:43:54.176 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-09T12:43:54.176 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-09T12:43:54.176 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.176 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.177 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:43:54.178 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-09T12:43:54.178 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-09T12:43:54.178 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-09T12:43:54.178 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-3.el9.x86_64 2024-03-09T12:43:54.178 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-09T12:43:54.178 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:43:54.178 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:43:54.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-test 2024-03-09T12:43:55.090 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:29 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:43:55.248 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:43:55.249 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:43:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-09T12:43:55.249 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:43:55.249 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:43:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 47 M 2024-03-09T12:43:55.249 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-09T12:43:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-09T12:43:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 163 k 2024-03-09T12:43:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-09T12:43:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-09T12:43:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-09T12:43:55.249 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:43:55.250 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:43:55.250 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:43:55.250 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-03-09T12:43:55.250 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:43:55.250 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 48 M 2024-03-09T12:43:55.250 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 205 M 2024-03-09T12:43:55.250 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:43:55.446 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-09T12:43:55.496 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2024-03-09T12:43:55.530 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-09T12:43:55.547 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): libcephsqlite-19.0.0-1920.g6b01fe95.el9. 646 kB/s | 163 kB 00:00 2024-03-09T12:43:55.756 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 831 kB/s | 187 kB 00:00 2024-03-09T12:43:57.847 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): ceph-test-19.0.0-1920.g6b01fe95.el9.x86_ 19 MB/s | 47 MB 00:02 2024-03-09T12:43:57.850 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:43:57.851 INFO:teuthology.orchestra.run.smithi162.stdout:Total 19 MB/s | 48 MB 00:02 2024-03-09T12:43:57.865 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:43:57.886 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:43:57.887 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:43:58.278 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:43:58.279 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:43:58.847 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:43:58.919 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-09T12:43:58.956 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-09T12:43:58.989 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-09T12:43:59.031 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-09T12:43:59.059 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:19.0.0-1920.g6b01fe95.el9.x86_64 5/6 2024-03-09T12:44:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1920.g6b01fe95.el9.x86_64 5/6 2024-03-09T12:44:03.666 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:19.0.0-1920.g6b01fe95.el9.x86_64 6/6 2024-03-09T12:44:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-test-2:19.0.0-1920.g6b01fe95.el9.x86_64 6/6 2024-03-09T12:44:06.174 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/6 2024-03-09T12:44:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:19.0.0-1920.g6b01fe95.el9.x86_64 2/6 2024-03-09T12:44:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-09T12:44:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-09T12:44:06.175 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-09T12:44:06.583 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-09T12:44:06.583 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:06.584 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:44:06.584 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:44:06.584 INFO:teuthology.orchestra.run.smithi162.stdout: jq-1.6-16.el9.x86_64 2024-03-09T12:44:06.584 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:44:06.584 INFO:teuthology.orchestra.run.smithi162.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-09T12:44:06.584 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-09T12:44:06.584 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-09T12:44:06.584 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:06.584 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:44:06.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph 2024-03-09T12:44:07.443 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:41 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:44:07.607 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:44:07.610 INFO:teuthology.orchestra.run.smithi162.stdout:========================================================================================= 2024-03-09T12:44:07.610 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-09T12:44:07.610 INFO:teuthology.orchestra.run.smithi162.stdout:========================================================================================= 2024-03-09T12:44:07.610 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:44:07.610 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 6.6 k 2024-03-09T12:44:07.610 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-09T12:44:07.610 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 2.1 M 2024-03-09T12:44:07.610 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 1.5 M 2024-03-09T12:44:07.611 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1920.g6b01fe95.el9 ceph-noarch 248 k 2024-03-09T12:44:07.611 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 4.7 M 2024-03-09T12:44:07.611 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 17 M 2024-03-09T12:44:07.611 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-09T12:44:07.611 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-09T12:44:07.611 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-09T12:44:07.611 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-09T12:44:07.611 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-09T12:44:07.611 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-09T12:44:07.611 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-09T12:44:07.611 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-09T12:44:07.611 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-09T12:44:07.611 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-09T12:44:07.612 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-09T12:44:07.613 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-09T12:44:07.613 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-09T12:44:07.613 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-09T12:44:07.613 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-09T12:44:07.613 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-09T12:44:07.613 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:07.613 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:44:07.613 INFO:teuthology.orchestra.run.smithi162.stdout:========================================================================================= 2024-03-09T12:44:07.613 INFO:teuthology.orchestra.run.smithi162.stdout:Install 36 Packages 2024-03-09T12:44:07.613 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:07.614 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 30 M 2024-03-09T12:44:07.614 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 107 M 2024-03-09T12:44:07.614 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:44:08.117 INFO:teuthology.orchestra.run.smithi162.stdout:(1/36): ceph-19.0.0-1920.g6b01fe95.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-03-09T12:44:08.334 INFO:teuthology.orchestra.run.smithi162.stdout:(2/36): ceph-mds-19.0.0-1920.g6b01fe95.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-03-09T12:44:08.659 INFO:teuthology.orchestra.run.smithi162.stdout:(3/36): ceph-mon-19.0.0-1920.g6b01fe95.el9.x86_ 8.7 MB/s | 4.7 MB 00:00 2024-03-09T12:44:08.735 INFO:teuthology.orchestra.run.smithi162.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1920.g6b01 3.2 MB/s | 248 kB 00:00 2024-03-09T12:44:08.843 INFO:teuthology.orchestra.run.smithi162.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-09T12:44:08.885 INFO:teuthology.orchestra.run.smithi162.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-09T12:44:08.927 INFO:teuthology.orchestra.run.smithi162.stdout:(7/36): ceph-mgr-19.0.0-1920.g6b01fe95.el9.x86_ 1.5 MB/s | 1.5 MB 00:00 2024-03-09T12:44:08.952 INFO:teuthology.orchestra.run.smithi162.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 471 kB/s | 32 kB 00:00 2024-03-09T12:44:08.969 INFO:teuthology.orchestra.run.smithi162.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-09T12:44:08.986 INFO:teuthology.orchestra.run.smithi162.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-09T12:44:09.012 INFO:teuthology.orchestra.run.smithi162.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-09T12:44:09.062 INFO:teuthology.orchestra.run.smithi162.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-09T12:44:09.087 INFO:teuthology.orchestra.run.smithi162.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-09T12:44:09.113 INFO:teuthology.orchestra.run.smithi162.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-09T12:44:09.138 INFO:teuthology.orchestra.run.smithi162.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-09T12:44:09.313 INFO:teuthology.orchestra.run.smithi162.stdout:(16/36): ceph-osd-19.0.0-1920.g6b01fe95.el9.x86 17 MB/s | 17 MB 00:00 2024-03-09T12:44:09.372 INFO:teuthology.orchestra.run.smithi162.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 95 kB/s | 22 kB 00:00 2024-03-09T12:44:09.447 INFO:teuthology.orchestra.run.smithi162.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 374 kB/s | 125 kB 00:00 2024-03-09T12:44:09.905 INFO:teuthology.orchestra.run.smithi162.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 81 kB/s | 43 kB 00:00 2024-03-09T12:44:09.931 INFO:teuthology.orchestra.run.smithi162.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 245 kB/s | 151 kB 00:00 2024-03-09T12:44:10.031 INFO:teuthology.orchestra.run.smithi162.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-09T12:44:10.140 INFO:teuthology.orchestra.run.smithi162.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 248 kB/s | 172 kB 00:00 2024-03-09T12:44:10.165 INFO:teuthology.orchestra.run.smithi162.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 132 kB/s | 18 kB 00:00 2024-03-09T12:44:10.290 INFO:teuthology.orchestra.run.smithi162.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 154 kB/s | 23 kB 00:00 2024-03-09T12:44:10.316 INFO:teuthology.orchestra.run.smithi162.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 129 kB/s | 19 kB 00:00 2024-03-09T12:44:10.399 INFO:teuthology.orchestra.run.smithi162.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-03-09T12:44:10.424 INFO:teuthology.orchestra.run.smithi162.stdout:(27/36): python3-cherrypy-18.6.1-2.el9.noarch.r 691 kB/s | 358 kB 00:00 2024-03-09T12:44:10.450 INFO:teuthology.orchestra.run.smithi162.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 346 kB/s | 46 kB 00:00 2024-03-09T12:44:10.558 INFO:teuthology.orchestra.run.smithi162.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 152 kB/s | 16 kB 00:00 2024-03-09T12:44:10.584 INFO:teuthology.orchestra.run.smithi162.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 428 kB/s | 79 kB 00:00 2024-03-09T12:44:10.692 INFO:teuthology.orchestra.run.smithi162.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 330 kB/s | 36 kB 00:00 2024-03-09T12:44:10.742 INFO:teuthology.orchestra.run.smithi162.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 489 kB/s | 90 kB 00:00 2024-03-09T12:44:10.876 INFO:teuthology.orchestra.run.smithi162.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 603 kB/s | 272 kB 00:00 2024-03-09T12:44:10.985 INFO:teuthology.orchestra.run.smithi162.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-03-09T12:44:11.035 INFO:teuthology.orchestra.run.smithi162.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 671 kB/s | 230 kB 00:00 2024-03-09T12:44:11.260 INFO:teuthology.orchestra.run.smithi162.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 826 kB/s | 427 kB 00:00 2024-03-09T12:44:11.266 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:44:11.266 INFO:teuthology.orchestra.run.smithi162.stdout:Total 8.1 MB/s | 30 MB 00:03 2024-03-09T12:44:11.291 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:44:11.334 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:44:11.334 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:44:11.754 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:44:11.754 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:44:13.075 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:44:13.137 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-09T12:44:13.170 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-09T12:44:13.198 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-09T12:44:13.268 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-09T12:44:13.322 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-09T12:44:13.357 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-09T12:44:13.448 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-09T12:44:13.527 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-09T12:44:13.568 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-09T12:44:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-09T12:44:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-09T12:44:13.740 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-09T12:44:13.789 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-09T12:44:13.848 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-09T12:44:14.045 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-09T12:44:14.104 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-09T12:44:14.150 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-09T12:44:14.185 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-09T12:44:14.216 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-09T12:44:14.292 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-09T12:44:14.326 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-09T12:44:14.359 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-09T12:44:14.399 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-09T12:44:14.434 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-09T12:44:14.477 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-09T12:44:14.567 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-09T12:44:14.610 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-09T12:44:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-09T12:44:14.819 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-09T12:44:14.936 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-09T12:44:15.093 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1920.g6b01fe95.el9. 31/36 2024-03-09T12:44:15.122 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:19.0.0-1920.g6b01fe95.el9.x86_64 32/36 2024-03-09T12:44:15.148 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1920.g6b01fe95.el9.x86_64 32/36 2024-03-09T12:44:15.148 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T12:44:15.148 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-09T12:44:15.149 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-09T12:44:15.149 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-09T12:44:15.149 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:16.582 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:19.0.0-1920.g6b01fe95.el9.x86_64 33/36 2024-03-09T12:44:16.609 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:19.0.0-1920.g6b01fe95.el9.x86_64 33/36 2024-03-09T12:44:16.609 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T12:44:16.609 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-09T12:44:16.609 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-09T12:44:16.609 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-09T12:44:16.609 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:17.058 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:19.0.0-1920.g6b01fe95.el9.x86_64 34/36 2024-03-09T12:44:17.081 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:19.0.0-1920.g6b01fe95.el9.x86_64 34/36 2024-03-09T12:44:17.081 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T12:44:17.082 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-09T12:44:17.082 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-09T12:44:17.082 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-09T12:44:17.082 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:17.283 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:19.0.0-1920.g6b01fe95.el9.x86_64 35/36 2024-03-09T12:44:17.306 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:19.0.0-1920.g6b01fe95.el9.x86_64 35/36 2024-03-09T12:44:17.307 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T12:44:17.307 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-09T12:44:17.307 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-09T12:44:17.307 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-09T12:44:17.307 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:17.373 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:19.0.0-1920.g6b01fe95.el9.x86_64 36/36 2024-03-09T12:44:19.164 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-2:19.0.0-1920.g6b01fe95.el9.x86_64 36/36 2024-03-09T12:44:19.164 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/36 2024-03-09T12:44:19.164 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:19.0.0-1920.g6b01fe95.el9.x86_64 2/36 2024-03-09T12:44:19.165 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:19.0.0-1920.g6b01fe95.el9.x86_64 3/36 2024-03-09T12:44:19.165 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:19.0.0-1920.g6b01fe95.el9.x86_64 4/36 2024-03-09T12:44:19.165 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:19.0.0-1920.g6b01fe95.el9.x86_64 5/36 2024-03-09T12:44:19.165 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1920.g6b01fe95.el9. 6/36 2024-03-09T12:44:19.165 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-09T12:44:19.165 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-09T12:44:19.165 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-09T12:44:19.165 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-09T12:44:19.165 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-09T12:44:19.165 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-09T12:44:19.165 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-09T12:44:19.165 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-09T12:44:19.165 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-09T12:44:19.166 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-09T12:44:19.166 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-09T12:44:19.166 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-09T12:44:19.166 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-09T12:44:19.166 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-09T12:44:19.166 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-09T12:44:19.166 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-09T12:44:19.166 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-09T12:44:19.166 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-09T12:44:19.166 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-09T12:44:19.167 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-09T12:44:19.168 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-09T12:44:19.168 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-09T12:44:19.168 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-09T12:44:19.168 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-09T12:44:19.168 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-09T12:44:19.168 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-09T12:44:19.168 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-09T12:44:19.168 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-09T12:44:19.168 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-09T12:44:20.112 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-09T12:44:20.112 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:20.112 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:44:20.112 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:44:20.112 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:44:20.112 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:44:20.112 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:19.0.0-1920.g6b01fe95.el9.noarch 2024-03-09T12:44:20.112 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:44:20.112 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-09T12:44:20.113 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-09T12:44:20.114 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-09T12:44:20.114 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-09T12:44:20.114 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-09T12:44:20.114 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-09T12:44:20.114 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2024-03-09T12:44:20.114 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-09T12:44:20.114 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-09T12:44:20.114 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-09T12:44:20.114 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-09T12:44:20.114 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-09T12:44:20.114 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-09T12:44:20.114 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-09T12:44:20.114 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-09T12:44:20.115 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-09T12:44:20.115 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-09T12:44:20.115 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-09T12:44:20.115 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:20.115 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:44:20.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base 2024-03-09T12:44:21.020 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:55 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:44:21.118 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-base-2:19.0.0-1920.g6b01fe95.el9.x86_64 is already installed. 2024-03-09T12:44:21.171 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:44:21.172 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-09T12:44:21.172 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:44:21.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install cephadm 2024-03-09T12:44:21.813 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:55 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:44:21.963 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:44:21.964 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:21.964 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-09T12:44:21.964 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:21.964 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:44:21.964 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:19.0.0-1920.g6b01fe95.el9 ceph-noarch 748 k 2024-03-09T12:44:21.964 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:21.964 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:44:21.965 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:21.965 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-09T12:44:21.965 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:21.965 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 748 k 2024-03-09T12:44:21.965 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 755 k 2024-03-09T12:44:21.965 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:44:22.240 INFO:teuthology.orchestra.run.smithi162.stdout:cephadm-19.0.0-1920.g6b01fe95.el9.noarch.rpm 2.7 MB/s | 748 kB 00:00 2024-03-09T12:44:22.240 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:44:22.241 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.6 MB/s | 748 kB 00:00 2024-03-09T12:44:22.241 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:44:22.246 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:44:22.246 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:44:22.252 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:44:22.253 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:44:22.327 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:44:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.0.0-1920.g6b01fe95.el9.noarch 1/1 2024-03-09T12:44:22.788 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:19.0.0-1920.g6b01fe95.el9.noarch 1/1 2024-03-09T12:44:23.192 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.0.0-1920.g6b01fe95.el9.noarch 1/1 2024-03-09T12:44:23.526 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:19.0.0-1920.g6b01fe95.el9.noarch 1/1 2024-03-09T12:44:23.526 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:23.526 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:44:23.526 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:19.0.0-1920.g6b01fe95.el9.noarch 2024-03-09T12:44:23.526 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:23.526 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:44:23.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-immutable-object-cache 2024-03-09T12:44:24.283 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:58 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:44:24.439 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:44:24.440 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-09T12:44:24.440 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:24.440 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:44:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 145 k 2024-03-09T12:44:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:24.440 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:44:24.440 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:24.440 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-09T12:44:24.440 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:24.440 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 145 k 2024-03-09T12:44:24.440 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 436 k 2024-03-09T12:44:24.440 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:44:24.956 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-immutable-object-cache-19.0.0-1920.g6b01fe 282 kB/s | 145 kB 00:00 2024-03-09T12:44:24.956 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:44:24.956 INFO:teuthology.orchestra.run.smithi162.stdout:Total 281 kB/s | 145 kB 00:00 2024-03-09T12:44:24.957 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:44:24.965 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:44:24.965 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:44:25.017 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:44:25.017 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:44:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:44:25.206 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1920.g6b01fe95. 1/1 2024-03-09T12:44:25.231 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1920.g6b01fe95. 1/1 2024-03-09T12:44:25.231 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T12:44:25.231 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-09T12:44:25.231 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:26.254 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1920.g6b01fe95. 1/1 2024-03-09T12:44:26.254 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:26.254 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:44:26.254 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:44:26.254 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:26.254 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:44:26.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr 2024-03-09T12:44:27.063 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:01 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:44:27.162 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-mgr-2:19.0.0-1920.g6b01fe95.el9.x86_64 is already installed. 2024-03-09T12:44:27.213 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:44:27.215 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-09T12:44:27.215 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:44:27.284 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-dashboard 2024-03-09T12:44:27.846 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:01 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:44:27.996 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:44:27.997 INFO:teuthology.orchestra.run.smithi162.stdout:====================================================================================== 2024-03-09T12:44:27.997 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-09T12:44:27.998 INFO:teuthology.orchestra.run.smithi162.stdout:====================================================================================== 2024-03-09T12:44:27.998 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:44:27.998 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1920.g6b01fe95.el9 ceph-noarch 1.8 M 2024-03-09T12:44:27.998 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-09T12:44:27.998 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-09T12:44:27.998 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1920.g6b01fe95.el9 ceph-noarch 34 k 2024-03-09T12:44:27.998 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1920.g6b01fe95.el9 ceph-noarch 16 k 2024-03-09T12:44:27.998 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-09T12:44:27.999 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-09T12:44:27.999 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-09T12:44:27.999 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-09T12:44:27.999 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-09T12:44:27.999 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-09T12:44:27.999 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-09T12:44:27.999 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-09T12:44:28.000 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:28.000 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:44:28.000 INFO:teuthology.orchestra.run.smithi162.stdout:====================================================================================== 2024-03-09T12:44:28.000 INFO:teuthology.orchestra.run.smithi162.stdout:Install 12 Packages 2024-03-09T12:44:28.000 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:28.001 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 6.9 M 2024-03-09T12:44:28.001 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 26 M 2024-03-09T12:44:28.002 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:44:28.402 INFO:teuthology.orchestra.run.smithi162.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1920.g6b 182 kB/s | 34 kB 00:00 2024-03-09T12:44:28.502 INFO:teuthology.orchestra.run.smithi162.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1920.g6b0 57 kB/s | 16 kB 00:00 2024-03-09T12:44:28.561 INFO:teuthology.orchestra.run.smithi162.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-09T12:44:28.586 INFO:teuthology.orchestra.run.smithi162.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-09T12:44:28.720 INFO:teuthology.orchestra.run.smithi162.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1920.g6b01fe9 3.6 MB/s | 1.8 MB 00:00 2024-03-09T12:44:28.845 INFO:teuthology.orchestra.run.smithi162.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-03-09T12:44:28.862 INFO:teuthology.orchestra.run.smithi162.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-03-09T12:44:28.954 INFO:teuthology.orchestra.run.smithi162.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 19 MB/s | 2.0 MB 00:00 2024-03-09T12:44:28.979 INFO:teuthology.orchestra.run.smithi162.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-03-09T12:44:28.996 INFO:teuthology.orchestra.run.smithi162.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 735 kB/s | 31 kB 00:00 2024-03-09T12:44:29.030 INFO:teuthology.orchestra.run.smithi162.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-09T12:44:29.339 INFO:teuthology.orchestra.run.smithi162.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.1 MB/s | 863 kB 00:00 2024-03-09T12:44:29.342 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:44:29.342 INFO:teuthology.orchestra.run.smithi162.stdout:Total 5.1 MB/s | 6.9 MB 00:01 2024-03-09T12:44:29.360 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:44:29.377 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:44:29.378 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:44:29.545 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:44:29.546 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:44:30.068 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:44:30.130 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-09T12:44:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-09T12:44:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-09T12:44:30.260 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-09T12:44:30.366 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-09T12:44:30.501 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-09T12:44:30.597 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-09T12:44:30.640 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-09T12:44:30.668 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-09T12:44:30.701 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1920.g6b01fe95.el9 10/12 2024-03-09T12:44:30.946 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1920.g6b01fe95.el 11/12 2024-03-09T12:44:30.984 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1920.g6b01fe95.el9.noa 12/12 2024-03-09T12:44:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1920.g6b01fe95.el9.noa 12/12 2024-03-09T12:44:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1920.g6b01fe95.el 1/12 2024-03-09T12:44:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1920.g6b01fe95.el9.noa 2/12 2024-03-09T12:44:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1920.g6b01fe95.el9 3/12 2024-03-09T12:44:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-09T12:44:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-09T12:44:31.773 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-09T12:44:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-09T12:44:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-09T12:44:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-09T12:44:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-09T12:44:31.774 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-09T12:44:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-09T12:44:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:32.144 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:44:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-09T12:44:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-grafana-dashboards-2:19.0.0-1920.g6b01fe95.el9.noarch 2024-03-09T12:44:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-dashboard-2:19.0.0-1920.g6b01fe95.el9.noarch 2024-03-09T12:44:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-prometheus-alerts-2:19.0.0-1920.g6b01fe95.el9.noarch 2024-03-09T12:44:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-09T12:44:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-09T12:44:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-09T12:44:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-09T12:44:32.144 INFO:teuthology.orchestra.run.smithi162.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-09T12:44:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-09T12:44:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-09T12:44:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-09T12:44:32.145 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:32.145 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:44:32.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-09T12:44:32.938 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:06 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:44:33.089 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:44:33.090 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-09T12:44:33.090 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-09T12:44:33.091 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-09T12:44:33.091 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:44:33.091 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1920.g6b01fe95.el9 ceph-noarch 7.4 M 2024-03-09T12:44:33.091 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-09T12:44:33.091 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-09T12:44:33.091 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-09T12:44:33.091 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-09T12:44:33.091 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-09T12:44:33.091 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-09T12:44:33.091 INFO:teuthology.orchestra.run.smithi162.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-09T12:44:33.091 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-09T12:44:33.091 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-09T12:44:33.091 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-09T12:44:33.092 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-09T12:44:33.092 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-09T12:44:33.092 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:33.092 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:44:33.092 INFO:teuthology.orchestra.run.smithi162.stdout:============================================================================================ 2024-03-09T12:44:33.092 INFO:teuthology.orchestra.run.smithi162.stdout:Install 12 Packages 2024-03-09T12:44:33.092 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:33.092 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 37 M 2024-03-09T12:44:33.093 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 209 M 2024-03-09T12:44:33.093 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:44:33.496 INFO:teuthology.orchestra.run.smithi162.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 222 kB/s | 32 kB 00:00 2024-03-09T12:44:33.529 INFO:teuthology.orchestra.run.smithi162.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 539 kB/s | 17 kB 00:00 2024-03-09T12:44:33.563 INFO:teuthology.orchestra.run.smithi162.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-09T12:44:33.663 INFO:teuthology.orchestra.run.smithi162.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.6 MB/s | 3.0 MB 00:00 2024-03-09T12:44:33.747 INFO:teuthology.orchestra.run.smithi162.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-09T12:44:33.839 INFO:teuthology.orchestra.run.smithi162.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 4.8 MB 00:00 2024-03-09T12:44:33.883 INFO:teuthology.orchestra.run.smithi162.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.6 MB/s | 217 kB 00:00 2024-03-09T12:44:34.181 INFO:teuthology.orchestra.run.smithi162.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-09T12:44:34.298 INFO:teuthology.orchestra.run.smithi162.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 803 kB 00:00 2024-03-09T12:44:34.424 INFO:teuthology.orchestra.run.smithi162.stdout:(10/12): ceph-mgr-diskprediction-local-19.0.0-1 6.9 MB/s | 7.4 MB 00:01 2024-03-09T12:44:34.458 INFO:teuthology.orchestra.run.smithi162.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 708 kB/s | 195 kB 00:00 2024-03-09T12:44:34.849 INFO:teuthology.orchestra.run.smithi162.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 446 kB/s | 245 kB 00:00 2024-03-09T12:44:34.851 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:44:34.851 INFO:teuthology.orchestra.run.smithi162.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-09T12:44:34.892 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:44:34.917 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:44:34.917 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:44:35.276 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:44:35.276 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:44:35.982 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:44:36.056 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-09T12:44:36.087 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-09T12:44:36.180 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-09T12:44:36.232 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-09T12:44:36.409 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-09T12:44:36.447 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-09T12:44:36.524 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-09T12:44:37.182 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-09T12:44:37.294 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-09T12:44:38.547 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-09T12:44:39.562 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-09T12:44:39.591 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1920.g6b01f 12/12 2024-03-09T12:44:42.173 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1920.g6b01f 12/12 2024-03-09T12:44:42.174 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1920.g6b01f 1/12 2024-03-09T12:44:42.174 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-09T12:44:42.174 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-09T12:44:42.174 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-09T12:44:42.174 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-09T12:44:42.174 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-09T12:44:42.174 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-09T12:44:42.174 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-09T12:44:42.174 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-09T12:44:42.174 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-09T12:44:42.174 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-09T12:44:42.601 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-09T12:44:42.601 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:42.601 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1920.g6b01fe95.el9.noarch 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:42.602 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:44:42.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-rook 2024-03-09T12:44:43.525 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:17 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:44:43.676 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:44:43.677 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-09T12:44:43.677 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-09T12:44:43.677 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-09T12:44:43.677 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:44:43.677 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook noarch 2:19.0.0-1920.g6b01fe95.el9 ceph-noarch 49 k 2024-03-09T12:44:43.677 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-09T12:44:43.677 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-09T12:44:43.677 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-09T12:44:43.677 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-09T12:44:43.677 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-09T12:44:43.677 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-09T12:44:43.678 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-09T12:44:43.678 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-09T12:44:43.678 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-09T12:44:43.678 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-09T12:44:43.678 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-09T12:44:43.678 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-09T12:44:43.678 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-09T12:44:43.678 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:43.678 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:44:43.678 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-09T12:44:43.678 INFO:teuthology.orchestra.run.smithi162.stdout:Install 13 Packages 2024-03-09T12:44:43.678 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:43.679 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 2.0 M 2024-03-09T12:44:43.679 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 26 M 2024-03-09T12:44:43.679 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:44:43.951 INFO:teuthology.orchestra.run.smithi162.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 189 kB/s | 24 kB 00:00 2024-03-09T12:44:43.976 INFO:teuthology.orchestra.run.smithi162.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2024-03-09T12:44:44.026 INFO:teuthology.orchestra.run.smithi162.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-09T12:44:44.052 INFO:teuthology.orchestra.run.smithi162.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-09T12:44:44.069 INFO:teuthology.orchestra.run.smithi162.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2024-03-09T12:44:44.095 INFO:teuthology.orchestra.run.smithi162.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-09T12:44:44.119 INFO:teuthology.orchestra.run.smithi162.stdout:(7/13): ceph-mgr-rook-19.0.0-1920.g6b01fe95.el9 166 kB/s | 49 kB 00:00 2024-03-09T12:44:44.203 INFO:teuthology.orchestra.run.smithi162.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 138 kB/s | 15 kB 00:00 2024-03-09T12:44:44.228 INFO:teuthology.orchestra.run.smithi162.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-03-09T12:44:44.337 INFO:teuthology.orchestra.run.smithi162.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 546 kB/s | 59 kB 00:00 2024-03-09T12:44:44.363 INFO:teuthology.orchestra.run.smithi162.stdout:(11/13): python3-google-auth-2.28.0-1.el9.noarc 834 kB/s | 203 kB 00:00 2024-03-09T12:44:44.421 INFO:teuthology.orchestra.run.smithi162.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-03-09T12:44:44.463 INFO:teuthology.orchestra.run.smithi162.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-09T12:44:44.465 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:44:44.465 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-03-09T12:44:44.466 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:44:44.478 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:44:44.478 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:44:44.583 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:44:44.583 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:44:45.104 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:44:45.278 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-09T12:44:45.331 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-09T12:44:45.386 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-09T12:44:45.424 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-09T12:44:45.465 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-09T12:44:45.559 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-09T12:44:45.668 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-09T12:44:45.733 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-09T12:44:46.456 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-09T12:44:46.506 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-09T12:44:46.540 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-09T12:44:46.588 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-09T12:44:46.610 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-rook-2:19.0.0-1920.g6b01fe95.el9.noarch 13/13 2024-03-09T12:44:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1920.g6b01fe95.el9.noarch 13/13 2024-03-09T12:44:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1920.g6b01fe95.el9.noarch 1/13 2024-03-09T12:44:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-09T12:44:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-09T12:44:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-09T12:44:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-09T12:44:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-09T12:44:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-09T12:44:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-09T12:44:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-09T12:44:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-09T12:44:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-09T12:44:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-09T12:44:47.772 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-09T12:44:47.772 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:47.772 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:44:47.772 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-rook-2:19.0.0-1920.g6b01fe95.el9.noarch 2024-03-09T12:44:47.772 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-09T12:44:47.772 INFO:teuthology.orchestra.run.smithi162.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-09T12:44:47.772 INFO:teuthology.orchestra.run.smithi162.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-09T12:44:47.773 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-09T12:44:47.773 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-09T12:44:47.773 INFO:teuthology.orchestra.run.smithi162.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-09T12:44:47.773 INFO:teuthology.orchestra.run.smithi162.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-09T12:44:47.773 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-09T12:44:47.773 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-09T12:44:47.773 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-09T12:44:47.774 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-09T12:44:47.774 INFO:teuthology.orchestra.run.smithi162.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-09T12:44:47.774 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:47.774 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:44:48.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr-cephadm 2024-03-09T12:44:48.577 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:22 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:44:48.727 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:44:48.728 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-09T12:44:48.728 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-09T12:44:48.728 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-09T12:44:48.728 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:44:48.728 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1920.g6b01fe95.el9 ceph-noarch 139 k 2024-03-09T12:44:48.728 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-09T12:44:48.728 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-09T12:44:48.728 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-09T12:44:48.728 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-09T12:44:48.729 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-09T12:44:48.729 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-09T12:44:48.729 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:48.729 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:44:48.729 INFO:teuthology.orchestra.run.smithi162.stdout:======================================================================================== 2024-03-09T12:44:48.729 INFO:teuthology.orchestra.run.smithi162.stdout:Install 6 Packages 2024-03-09T12:44:48.729 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:48.729 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 6.8 M 2024-03-09T12:44:48.730 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 33 M 2024-03-09T12:44:48.730 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:44:49.075 INFO:teuthology.orchestra.run.smithi162.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-09T12:44:49.142 INFO:teuthology.orchestra.run.smithi162.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 758 kB/s | 248 kB 00:00 2024-03-09T12:44:49.167 INFO:teuthology.orchestra.run.smithi162.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1920.g6b01fe95.e 394 kB/s | 139 kB 00:00 2024-03-09T12:44:49.359 INFO:teuthology.orchestra.run.smithi162.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 266 kB/s | 58 kB 00:00 2024-03-09T12:44:49.385 INFO:teuthology.orchestra.run.smithi162.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 235 kB/s | 51 kB 00:00 2024-03-09T12:44:49.501 INFO:teuthology.orchestra.run.smithi162.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-03-09T12:44:49.502 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:44:49.502 INFO:teuthology.orchestra.run.smithi162.stdout:Total 8.8 MB/s | 6.8 MB 00:00 2024-03-09T12:44:49.514 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:44:49.525 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:44:49.525 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:44:49.616 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:44:49.617 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:44:49.925 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:44:50.101 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-09T12:44:50.161 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-09T12:44:50.685 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-09T12:44:50.772 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-09T12:44:50.851 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-09T12:44:50.874 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1920.g6b01fe95.el9.noarch 6/6 2024-03-09T12:44:51.677 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1920.g6b01fe95.el9.noarch 6/6 2024-03-09T12:44:51.677 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1920.g6b01fe95.el9.noarch 1/6 2024-03-09T12:44:51.677 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-09T12:44:51.677 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-09T12:44:51.678 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-09T12:44:51.678 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-09T12:44:51.987 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-09T12:44:51.987 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:51.987 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:44:51.987 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-cephadm-2:19.0.0-1920.g6b01fe95.el9.noarch 2024-03-09T12:44:51.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-09T12:44:51.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-09T12:44:51.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-09T12:44:51.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-09T12:44:51.987 INFO:teuthology.orchestra.run.smithi162.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-09T12:44:51.988 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:51.988 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:44:52.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-fuse 2024-03-09T12:44:52.809 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:26 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:44:52.960 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:44:52.965 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:52.965 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-09T12:44:52.966 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:52.966 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:44:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 888 k 2024-03-09T12:44:52.966 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-09T12:44:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-09T12:44:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:52.966 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:44:52.966 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:52.966 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-03-09T12:44:52.966 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:52.967 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 967 k 2024-03-09T12:44:52.967 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.8 M 2024-03-09T12:44:52.967 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:44:53.088 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 631 kB/s | 79 kB 00:00 2024-03-09T12:44:53.287 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): ceph-fuse-19.0.0-1920.g6b01fe95.el9.x86_ 2.7 MB/s | 888 kB 00:00 2024-03-09T12:44:53.289 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:44:53.290 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.9 MB/s | 967 kB 00:00 2024-03-09T12:44:53.290 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:44:53.310 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:44:53.310 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:44:53.389 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:44:53.389 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:44:53.614 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:44:53.756 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-09T12:44:53.831 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:19.0.0-1920.g6b01fe95.el9.x86_64 2/2 2024-03-09T12:44:54.586 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1920.g6b01fe95.el9.x86_64 2/2 2024-03-09T12:44:54.586 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/2 2024-03-09T12:44:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-09T12:44:54.921 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:54.921 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:44:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:19.0.0-1920.g6b01fe95.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-09T12:44:54.922 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:54.922 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:44:55.221 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-volume 2024-03-09T12:44:55.801 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:29 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:44:55.951 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:44:55.951 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:55.952 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-03-09T12:44:55.952 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:55.952 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:44:55.952 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume noarch 2:19.0.0-1920.g6b01fe95.el9 ceph-noarch 264 k 2024-03-09T12:44:55.952 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-03-09T12:44:55.952 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-09T12:44:55.952 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:55.952 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:44:55.952 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:55.952 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-03-09T12:44:55.952 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:55.953 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 332 k 2024-03-09T12:44:55.953 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 1.5 M 2024-03-09T12:44:55.953 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:44:56.076 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 559 kB/s | 69 kB 00:00 2024-03-09T12:44:56.326 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): ceph-volume-19.0.0-1920.g6b01fe95.el9.no 707 kB/s | 264 kB 00:00 2024-03-09T12:44:56.327 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:44:56.328 INFO:teuthology.orchestra.run.smithi162.stdout:Total 885 kB/s | 332 kB 00:00 2024-03-09T12:44:56.328 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:44:56.337 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:44:56.337 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:44:56.378 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:44:56.379 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:44:56.532 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:44:56.710 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-09T12:44:56.746 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-volume-2:19.0.0-1920.g6b01fe95.el9.noarch 2/2 2024-03-09T12:44:56.762 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-volume-2:19.0.0-1920.g6b01fe95.el9.noarch 2/2 2024-03-09T12:44:56.762 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T12:44:56.762 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-09T12:44:56.762 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:57.487 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-volume-2:19.0.0-1920.g6b01fe95.el9.noarch 1/2 2024-03-09T12:44:57.836 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-09T12:44:57.836 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:57.836 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:44:57.836 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-volume-2:19.0.0-1920.g6b01fe95.el9.noarch 2024-03-09T12:44:57.836 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-09T12:44:57.836 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:57.836 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:44:58.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados-devel 2024-03-09T12:44:58.757 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:32 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:44:58.907 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:44:58.908 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:58.908 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-09T12:44:58.909 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:58.909 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:44:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 126 k 2024-03-09T12:44:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:58.909 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:44:58.909 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:44:58.909 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-09T12:44:58.909 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:44:58.909 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 126 k 2024-03-09T12:44:58.909 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 456 k 2024-03-09T12:44:58.909 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:44:59.216 INFO:teuthology.orchestra.run.smithi162.stdout:librados-devel-19.0.0-1920.g6b01fe95.el9.x86_64 412 kB/s | 126 kB 00:00 2024-03-09T12:44:59.217 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:44:59.217 INFO:teuthology.orchestra.run.smithi162.stdout:Total 410 kB/s | 126 kB 00:00 2024-03-09T12:44:59.217 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:44:59.223 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:44:59.223 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:44:59.279 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:44:59.279 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:44:59.428 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:44:59.539 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:44:59.994 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados-devel-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:00.328 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:00.329 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:45:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:45:00.329 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:00.329 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:45:00.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs2 2024-03-09T12:45:01.156 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:35 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:45:01.256 INFO:teuthology.orchestra.run.smithi162.stdout:Package libcephfs2-2:19.0.0-1920.g6b01fe95.el9.x86_64 is already installed. 2024-03-09T12:45:01.310 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:45:01.311 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-09T12:45:01.311 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:45:01.383 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs-devel 2024-03-09T12:45:01.946 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:35 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:45:02.097 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:45:02.098 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:45:02.098 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-09T12:45:02.098 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:45:02.098 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:45:02.098 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 31 k 2024-03-09T12:45:02.098 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:02.098 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:45:02.098 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:45:02.098 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-09T12:45:02.099 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:02.099 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 31 k 2024-03-09T12:45:02.099 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 140 k 2024-03-09T12:45:02.099 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:45:02.337 INFO:teuthology.orchestra.run.smithi162.stdout:libcephfs-devel-19.0.0-1920.g6b01fe95.el9.x86_6 131 kB/s | 31 kB 00:00 2024-03-09T12:45:02.338 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:45:02.338 INFO:teuthology.orchestra.run.smithi162.stdout:Total 130 kB/s | 31 kB 00:00 2024-03-09T12:45:02.338 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:45:02.341 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:45:02.341 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:45:02.371 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:45:02.371 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:45:02.493 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:45:02.577 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:02.960 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:03.235 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:03.235 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:03.235 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:45:03.235 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:45:03.236 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:03.236 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:45:03.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados2 2024-03-09T12:45:04.002 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:38 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:45:04.099 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:19.0.0-1920.g6b01fe95.el9.x86_64 is already installed. 2024-03-09T12:45:04.151 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:45:04.152 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-09T12:45:04.152 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:45:04.202 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librbd1 2024-03-09T12:45:04.761 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:38 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:45:04.859 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:19.0.0-1920.g6b01fe95.el9.x86_64 is already installed. 2024-03-09T12:45:04.911 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:45:04.912 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-09T12:45:04.912 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:45:04.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rados 2024-03-09T12:45:05.534 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:39 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:45:05.631 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rados-2:19.0.0-1920.g6b01fe95.el9.x86_64 is already installed. 2024-03-09T12:45:05.682 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:45:05.683 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-09T12:45:05.683 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:45:05.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rgw 2024-03-09T12:45:06.301 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:40 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:45:06.398 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rgw-2:19.0.0-1920.g6b01fe95.el9.x86_64 is already installed. 2024-03-09T12:45:06.449 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:45:06.450 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-09T12:45:06.450 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:45:06.503 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-cephfs 2024-03-09T12:45:07.068 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:41 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:45:07.165 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-cephfs-2:19.0.0-1920.g6b01fe95.el9.x86_64 is already installed. 2024-03-09T12:45:07.217 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:45:07.218 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-09T12:45:07.218 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:45:07.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rbd 2024-03-09T12:45:07.833 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:41 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:45:07.929 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rbd-2:19.0.0-1920.g6b01fe95.el9.x86_64 is already installed. 2024-03-09T12:45:07.981 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:45:07.982 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-03-09T12:45:07.982 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:45:08.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-fuse 2024-03-09T12:45:08.593 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:42 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:45:08.743 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:45:08.744 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:45:08.744 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-03-09T12:45:08.744 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:45:08.744 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:45:08.744 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 85 k 2024-03-09T12:45:08.744 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:08.744 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:45:08.745 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:45:08.745 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-09T12:45:08.745 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:08.745 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 85 k 2024-03-09T12:45:08.745 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 231 k 2024-03-09T12:45:08.745 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:45:08.949 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-fuse-19.0.0-1920.g6b01fe95.el9.x86_64.rpm 417 kB/s | 85 kB 00:00 2024-03-09T12:45:08.950 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:45:08.950 INFO:teuthology.orchestra.run.smithi162.stdout:Total 415 kB/s | 85 kB 00:00 2024-03-09T12:45:08.950 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:45:08.958 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:45:08.958 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:45:09.013 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:45:09.013 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:45:09.150 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:45:09.248 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:09.673 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:10.015 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:10.015 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:10.015 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:45:10.015 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:45:10.015 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:10.015 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:45:10.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-mirror 2024-03-09T12:45:10.866 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:44 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:45:11.019 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:45:11.020 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:45:11.020 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-03-09T12:45:11.020 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:45:11.021 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:45:11.021 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 3.1 M 2024-03-09T12:45:11.021 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:11.021 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:45:11.021 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:45:11.021 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-09T12:45:11.021 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:11.021 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 3.1 M 2024-03-09T12:45:11.021 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 13 M 2024-03-09T12:45:11.021 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:45:12.837 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-mirror-19.0.0-1920.g6b01fe95.el9.x86_64.rpm 1.7 MB/s | 3.1 MB 00:01 2024-03-09T12:45:12.838 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:45:12.838 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.7 MB/s | 3.1 MB 00:01 2024-03-09T12:45:12.839 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:45:12.850 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:45:12.850 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:45:12.926 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:45:12.927 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:45:13.352 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:45:13.413 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:13.439 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:13.439 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-09T12:45:13.439 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-09T12:45:13.439 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-09T12:45:13.439 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-09T12:45:13.439 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:14.642 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:14.642 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:14.642 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:45:14.642 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:45:14.642 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:14.642 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:45:14.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-nbd 2024-03-09T12:45:15.461 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:49 ago on Sat 09 Mar 2024 12:43:26 PM UTC. 2024-03-09T12:45:15.613 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-03-09T12:45:15.614 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:45:15.614 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-03-09T12:45:15.614 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:45:15.614 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-03-09T12:45:15.614 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:19.0.0-1920.g6b01fe95.el9 ceph 170 k 2024-03-09T12:45:15.614 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:15.614 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-03-09T12:45:15.614 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-03-09T12:45:15.614 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-03-09T12:45:15.615 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:15.615 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 170 k 2024-03-09T12:45:15.615 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 490 k 2024-03-09T12:45:15.615 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-03-09T12:45:16.300 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-nbd-19.0.0-1920.g6b01fe95.el9.x86_64.rpm 249 kB/s | 170 kB 00:00 2024-03-09T12:45:16.301 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-03-09T12:45:16.301 INFO:teuthology.orchestra.run.smithi162.stdout:Total 249 kB/s | 170 kB 00:00 2024-03-09T12:45:16.301 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-03-09T12:45:16.309 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-03-09T12:45:16.309 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-03-09T12:45:16.367 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-03-09T12:45:16.367 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-03-09T12:45:16.520 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-03-09T12:45:16.622 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:17.123 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:17.466 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:19.0.0-1920.g6b01fe95.el9.x86_64 1/1 2024-03-09T12:45:17.466 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:17.466 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-03-09T12:45:17.466 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:19.0.0-1920.g6b01fe95.el9.x86_64 2024-03-09T12:45:17.466 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-03-09T12:45:17.466 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-03-09T12:45:17.687 DEBUG:teuthology.parallel:result is None 2024-03-09T12:45:17.687 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6b01fe95f3aa686f1951a0c5828907a52930b92a 2024-03-09T12:45:17.913 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-09T12:45:17.934 INFO:teuthology.orchestra.run.smithi162.stdout:19.0.0-1920.g6b01fe95.el9 2024-03-09T12:45:17.934 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1920.g6b01fe95.el9 2024-03-09T12:45:17.934 INFO:teuthology.task.install:The correct ceph version 19.0.0-1920.g6b01fe95 is installed. 2024-03-09T12:45:17.936 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-09T12:45:17.936 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-09T12:45:17.936 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-09T12:45:18.002 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-09T12:45:18.002 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-09T12:45:18.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-03-09T12:45:18.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-09T12:45:18.141 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-09T12:45:18.141 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-09T12:45:18.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-09T12:45:18.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-09T12:45:18.280 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-09T12:45:18.280 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-09T12:45:18.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-03-09T12:45:18.365 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-09T12:45:18.432 INFO:teuthology.run_tasks:Running task workunit... 2024-03-09T12:45:18.449 INFO:tasks.workunit:Pulling workunits from ref 6b01fe95f3aa686f1951a0c5828907a52930b92a 2024-03-09T12:45:18.450 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-09T12:45:18.450 INFO:tasks.workunit:timeout=3h 2024-03-09T12:45:18.450 INFO:tasks.workunit:cleanup=True 2024-03-09T12:45:18.450 DEBUG:teuthology.orchestra.run.smithi162:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-09T12:45:18.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T12:45:18.487 INFO:teuthology.orchestra.run.smithi162.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-09T12:45:18.487 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-09T12:45:18.541 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-09T12:45:18.541 DEBUG:teuthology.orchestra.run.smithi162:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-09T12:45:18.596 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 6b01fe95f3aa686f1951a0c5828907a52930b92a 2024-03-09T12:45:18.651 INFO:tasks.workunit.client.0.smithi162.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-09T12:46:05.326 INFO:tasks.workunit.client.0.smithi162.stderr:Note: switching to '6b01fe95f3aa686f1951a0c5828907a52930b92a'. 2024-03-09T12:46:05.326 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr:state without impacting any branches by switching back to a branch. 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr: git switch -c 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr:Or undo this operation with: 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr: git switch - 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-03-09T12:46:05.327 INFO:tasks.workunit.client.0.smithi162.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-09T12:46:05.328 INFO:tasks.workunit.client.0.smithi162.stderr: 2024-03-09T12:46:05.328 INFO:tasks.workunit.client.0.smithi162.stderr:HEAD is now at 6b01fe95f3a Merge branch 'wip-refactor-ec-get_chunk_size' of https://github.com/rzarzynski/ceph into wip-yuri8-testing-2024-03-06-1329 2024-03-09T12:46:05.331 DEBUG:teuthology.orchestra.run.smithi162:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-09T12:46:05.387 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-03-09T12:46:05.387 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-09T12:46:05.442 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2024-03-09T12:46:05.442 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2024-03-09T12:46:05.442 DEBUG:teuthology.orchestra.run.smithi162:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6b01fe95f3aa686f1951a0c5828907a52930b92a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2024-03-09T12:46:05.510 INFO:tasks.workunit.client.0.smithi162.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-09T12:46:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-09T12:46:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T12:46:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T12:46:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-09T12:46:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-09T12:46:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-09T12:46:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-09T12:46:05.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-09T12:46:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-09T12:46:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-mapper 2024-03-09T12:46:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2024-03-09T12:46:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2024-03-09T12:46:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2024-03-09T12:46:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2024-03-09T12:46:05.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2024-03-09T12:46:05.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2024-03-09T12:46:05.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=7c37a482-1a91-41cc-9740-5a8ed6639fec --auth-supported=none ' 2024-03-09T12:46:05.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2024-03-09T12:46:05.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-09T12:46:05.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2024-03-09T12:46:05.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-09T12:46:05.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2024-03-09T12:46:05.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2024-03-09T12:46:05.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2024-03-09T12:46:05.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-mapper 2024-03-09T12:46:05.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-mapper 2024-03-09T12:46:05.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-09T12:46:05.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T12:46:05.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-09T12:46:05.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:46:05.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:46:05.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:46:05.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:46:05.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:46:05.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:46:05.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T12:46:05.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T12:46:05.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T12:46:05.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T12:46:05.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T12:46:05.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T12:46:05.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:46:05.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T12:46:05.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T12:46:05.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:46:05.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T12:46:05.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T12:46:05.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-09T12:46:05.530 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T12:46:05.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:46:05.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28754 2024-03-09T12:46:05.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28754 2024-03-09T12:46:05.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T12:46:05.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T12:46:05.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-mapper 2024-03-09T12:46:05.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T12:46:05.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:46:05.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28754 2024-03-09T12:46:05.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.28754 2024-03-09T12:46:05.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T12:46:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-09T12:46:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-09T12:46:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T12:46:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2024-03-09T12:46:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2024-03-09T12:46:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2024-03-09T12:46:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2024-03-09T12:46:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2024-03-09T12:46:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2024-03-09T12:46:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2024-03-09T12:46:05.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2024-03-09T12:46:05.536 INFO:tasks.workunit.client.0.smithi162.stdout:Dir: td/osd-mapper 2024-03-09T12:46:05.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2024-03-09T12:46:05.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-mapper 2024-03-09T12:46:05.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-03-09T12:46:05.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-03-09T12:46:05.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=4 2024-03-09T12:46:05.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=test 2024-03-09T12:46:05.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=test 2024-03-09T12:46:05.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local extra_pars= 2024-03-09T12:46:05.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-03-09T12:46:05.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-03-09T12:46:05.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-03-09T12:46:07.215 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 a003eb2d-f7f0-4efb-bce0-a5667969f716 2024-03-09T12:46:07.445 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:46:07.538 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:07.539+0000 7f1e4f60d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:46:07.540 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:07.541+0000 7f1e4f60d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:46:07.541 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:07.542+0000 7f1e4f60d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:46:07.541 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:07.542+0000 7f1e4f60d640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-09T12:46:07.541 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:07.542+0000 7f1e4f60d640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-09T12:46:07.542 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:07.542+0000 7f1e4f60d640 -1 bluestore(td/osd-mapper/0/block) _read_bdev_label failed to open td/osd-mapper/0/block: (2) No such file or directory 2024-03-09T12:46:07.614 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:07.614+0000 7f1e4f60d640 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2024-03-09T12:46:13.079 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T12:46:13.431 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T12:46:13.459 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:13.460+0000 7f83fd48b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:46:13.461 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:13.462+0000 7f83fd48b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:46:13.462 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:13.463+0000 7f83fd48b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:46:13.705 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:46:13.986 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:13.987+0000 7f83fd48b640 -1 Falling back to public interface 2024-03-09T12:46:14.978 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:46:16.257 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:46:16.311 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:16.312+0000 7f83fd48b640 -1 osd.0 0 log_to_monitors true 2024-03-09T12:46:17.587 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:46:18.875 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T12:46:20.186 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T12:46:21.470 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T12:46:22.753 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T12:46:24.045 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T12:46:25.323 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T12:46:26.605 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T12:46:27.885 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T12:46:29.727 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T12:46:31.407 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T12:46:33.035 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T12:46:34.699 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T12:46:36.385 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T12:46:37.740 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T12:46:38.022 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3685797745,v1:127.0.0.1:6803/3685797745] [v2:127.0.0.1:6804/3685797745,v1:127.0.0.1:6805/3685797745] exists,up a003eb2d-f7f0-4efb-bce0-a5667969f716 2024-03-09T12:46:38.025 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 ec752f56-3598-426c-92ef-88ec7433c993 2024-03-09T12:46:39.923 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:46:39.968 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:39.968+0000 7efc17caa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:46:40.009 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:40.010+0000 7efc17caa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:46:40.010 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:40.011+0000 7efc17caa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:46:40.011 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:40.011+0000 7efc17caa640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-09T12:46:40.011 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:40.011+0000 7efc17caa640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-09T12:46:40.011 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:40.011+0000 7efc17caa640 -1 bluestore(td/osd-mapper/1/block) _read_bdev_label failed to open td/osd-mapper/1/block: (2) No such file or directory 2024-03-09T12:46:40.099 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:40.100+0000 7efc17caa640 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2024-03-09T12:46:45.841 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T12:46:46.193 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T12:46:46.221 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:46.223+0000 7fcfd9573640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:46:46.223 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:46.225+0000 7fcfd9573640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:46:46.225 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:46.225+0000 7fcfd9573640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:46:46.468 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:46:46.746 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:46.747+0000 7fcfd9573640 -1 Falling back to public interface 2024-03-09T12:46:47.740 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:46:49.020 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:46:49.032 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:46:49.033+0000 7fcfd9573640 -1 osd.1 0 log_to_monitors true 2024-03-09T12:46:50.341 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:46:51.662 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T12:46:53.050 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T12:46:54.357 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T12:46:55.654 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T12:46:56.953 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T12:46:58.230 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T12:46:59.511 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T12:47:00.861 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T12:47:02.648 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T12:47:04.246 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T12:47:05.793 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T12:47:07.179 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T12:47:09.129 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T12:47:11.243 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T12:47:12.653 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T12:47:12.930 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3093246720,v1:127.0.0.1:6811/3093246720] [v2:127.0.0.1:6812/3093246720,v1:127.0.0.1:6813/3093246720] exists,up ec752f56-3598-426c-92ef-88ec7433c993 2024-03-09T12:47:12.934 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 58e770f2-9069-469f-83f9-86bcc7265059 2024-03-09T12:47:13.312 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:47:13.355 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:47:13.355+0000 7fad40c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:47:13.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:47:13.389+0000 7fad40c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:47:13.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:47:13.390+0000 7fad40c1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:47:13.390 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:47:13.390+0000 7fad40c1b640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-09T12:47:13.390 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:47:13.390+0000 7fad40c1b640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-09T12:47:13.390 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:47:13.390+0000 7fad40c1b640 -1 bluestore(td/osd-mapper/2/block) _read_bdev_label failed to open td/osd-mapper/2/block: (2) No such file or directory 2024-03-09T12:47:13.474 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:47:13.474+0000 7fad40c1b640 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2024-03-09T12:47:19.469 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T12:47:19.819 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T12:47:19.847 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:47:19.847+0000 7ff9b67f8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:47:19.849 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:47:19.849+0000 7ff9b67f8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:47:19.850 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:47:19.850+0000 7ff9b67f8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:47:20.089 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:47:20.380 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:47:20.381+0000 7ff9b67f8640 -1 Falling back to public interface 2024-03-09T12:47:21.358 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:47:22.638 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:47:22.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:47:22.754+0000 7ff9b67f8640 -1 osd.2 0 log_to_monitors true 2024-03-09T12:47:23.946 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:47:25.228 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T12:47:26.534 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T12:47:27.835 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T12:47:29.116 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T12:47:30.400 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T12:47:31.755 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T12:47:33.040 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T12:47:34.357 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T12:47:36.237 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T12:47:37.804 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T12:47:39.251 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T12:47:40.819 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T12:47:42.805 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T12:47:44.524 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T12:47:45.809 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T12:47:46.097 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/260555359,v1:127.0.0.1:6819/260555359] [v2:127.0.0.1:6820/260555359,v1:127.0.0.1:6821/260555359] exists,up 58e770f2-9069-469f-83f9-86bcc7265059 2024-03-09T12:47:46.486 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'test' created 2024-03-09T12:47:48.462 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836498 2024-03-09T12:47:48.765 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672972 2024-03-09T12:47:49.039 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509448 2024-03-09T12:47:51.588 INFO:tasks.workunit.client.0.smithi162.stdout:standard_scrub_cluster: dbg: test pool is test 1 2024-03-09T12:47:51.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2024-03-09T12:47:51.724 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0: { 2024-03-09T12:47:51.725 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-09T12:47:51.725 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:47:51.738 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1: { 2024-03-09T12:47:51.739 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-09T12:47:51.739 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:47:51.752 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2: { 2024-03-09T12:47:51.752 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_stats_update_period_not_scrubbing = '' (not observed, change may require restart) " 2024-03-09T12:47:51.752 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:47:51.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2024-03-09T12:47:51.866 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0: { 2024-03-09T12:47:51.866 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-09T12:47:51.866 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:47:51.880 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1: { 2024-03-09T12:47:51.880 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-09T12:47:51.880 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:47:51.894 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2: { 2024-03-09T12:47:51.894 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_stats_update_period_scrubbing = '' (not observed, change may require restart) " 2024-03-09T12:47:51.894 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:47:51.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2024-03-09T12:47:51.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2024-03-09T12:47:51.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2024-03-09T12:47:51.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2024-03-09T12:47:51.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2024-03-09T12:47:51.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-09T12:47:51.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-09T12:47:52.134 INFO:tasks.workunit.client.0.smithi162.stdout:created pool test snap snap01 2024-03-09T12:47:52.377 INFO:tasks.workunit.client.0.smithi162.stdout:created pool test snap snap02 2024-03-09T12:47:52.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2024-03-09T12:47:52.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-09T12:47:52.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-09T12:47:52.813 INFO:tasks.workunit.client.0.smithi162.stdout:created pool test snap snap13 2024-03-09T12:47:52.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2024-03-09T12:47:52.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2024-03-09T12:47:52.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2024-03-09T12:47:53.266 INFO:tasks.workunit.client.0.smithi162.stdout:created pool test snap snap24 2024-03-09T12:47:53.450 INFO:tasks.workunit.client.0.smithi162.stdout:created pool test snap snap25 2024-03-09T12:47:53.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 14058 2024-03-09T12:47:53.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2024-03-09T12:47:53.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-09T12:47:53.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2024-03-09T12:47:54.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2024-03-09T12:47:54.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-09T12:47:54.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2024-03-09T12:47:54.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2024-03-09T12:47:54.298 INFO:tasks.workunit.client.0.smithi162.stdout:pgid is 1.3 (primary: osd.1) 2024-03-09T12:47:54.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2024-03-09T12:47:54.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2024-03-09T12:47:54.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:295: set_query_debug: local pgid=1.3 2024-03-09T12:47:54.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: ceph pg dump pgs_brief 2024-03-09T12:47:54.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2024-03-09T12:47:54.550 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs_brief 2024-03-09T12:47:54.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:297: set_query_debug: local prim_osd=1 2024-03-09T12:47:54.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:299: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2024-03-09T12:47:54.563 INFO:tasks.workunit.client.0.smithi162.stdout:Setting scrub debug data. Primary for 1.3 is 1 2024-03-09T12:47:54.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: get_asok_path osd.1 2024-03-09T12:47:54.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T12:47:54.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T12:47:54.564 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T12:47:54.564 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:47:54.565 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28754 2024-03-09T12:47:54.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.28754/ceph-osd.1.asok 2024-03-09T12:47:54.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: CEPH_ARGS= 2024-03-09T12:47:54.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:300: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.28754/ceph-osd.1.asok scrubdebug 1.3 set sessions 2024-03-09T12:47:54.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T12:47:54.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-03-09T12:47:54.682 INFO:tasks.workunit.client.0.smithi162.stdout:{"success":true}{ 2024-03-09T12:47:54.683 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "objxxx", 2024-03-09T12:47:54.683 INFO:tasks.workunit.client.0.smithi162.stdout: "clones": [ 2024-03-09T12:47:54.683 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:47:54.683 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 2, 2024-03-09T12:47:54.683 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [ 2024-03-09T12:47:54.683 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:47:54.683 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 1, 2024-03-09T12:47:54.683 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap01" 2024-03-09T12:47:54.683 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T12:47:54.683 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:47:54.683 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 2, 2024-03-09T12:47:54.683 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap02" 2024-03-09T12:47:54.683 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6, 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: "overlaps": [] 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 3, 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [ 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 3, 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap13" 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6, 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: "overlaps": [] 2024-03-09T12:47:54.684 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 5, 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [ 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 4, 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap24" 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 5, 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap25" 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 5, 2024-03-09T12:47:54.685 INFO:tasks.workunit.client.0.smithi162.stdout: "overlaps": [] 2024-03-09T12:47:54.686 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T12:47:54.686 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:47:54.686 INFO:tasks.workunit.client.0.smithi162.stdout: "id": "head", 2024-03-09T12:47:54.686 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [], 2024-03-09T12:47:54.686 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6 2024-03-09T12:47:54.686 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-03-09T12:47:54.686 INFO:tasks.workunit.client.0.smithi162.stdout: ] 2024-03-09T12:47:54.686 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:47:54.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-09T12:47:54.778 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T12:47:54.778 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": true, 2024-03-09T12:47:54.778 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2024-03-09T12:47:54.778 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2024-03-09T12:47:54.778 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:47:54.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2024-03-09T12:47:57.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-09T12:47:58.052 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T12:47:58.053 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 2 0 0 0 0 12 0 0 3 0 3 active+clean 2024-03-09T12:47:47.815736+0000 24'3 24:26 [1,2,0] 1 [1,2,0] 1 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:47:58.053 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:47:47.704554+0000 0'0 24:23 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:47:58.053 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:47:47.553398+0000 0'0 27:32 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:47:58.053 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:47:47.704536+0000 0'0 24:25 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:47:58.053 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:47:58.053 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T12:47:58.053 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:47:58.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2024-03-09T12:47:58.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-09T12:47:58.318 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T12:47:58.318 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 2 0 0 0 0 12 0 0 3 0 3 active+clean 2024-03-09T12:47:47.815736+0000 24'3 24:26 [1,2,0] 1 [1,2,0] 1 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:47:58.318 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:47:47.704554+0000 0'0 24:23 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:47:58.318 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:47:47.553398+0000 0'0 27:32 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:47:58.318 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:47:47.704536+0000 0'0 24:25 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:47:58.318 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:47:58.318 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T12:47:58.319 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:47:58.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2024-03-09T12:47:58.705 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T12:47:58.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2024-03-09T12:47:59.107 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T12:47:59.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2024-03-09T12:48:04.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2024-03-09T12:48:04.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-03-09T12:48:04.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:48:04.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:48:04.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:48:04.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:48:04.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:48:04.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:48:04.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T12:48:04.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-09T12:48:04.457 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.456+0000 7fce17248980 1 bdev(0x562361252000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-09T12:48:04.457 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.457+0000 7fce17248980 0 bdev(0x562361252000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-09T12:48:04.457 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.457+0000 7fce17248980 1 bdev(0x562361252000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:04.457 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.457+0000 7fce17248980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-09T12:48:04.457 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.457+0000 7fce17248980 1 bdev(0x562361252700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-09T12:48:04.457 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.457+0000 7fce17248980 0 bdev(0x562361252700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-09T12:48:04.457 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.457+0000 7fce17248980 1 bdev(0x562361252700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:04.457 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.457+0000 7fce17248980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-09T12:48:04.457 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.457+0000 7fce17248980 1 bluefs mount 2024-03-09T12:48:04.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.479+0000 7fce17248980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T12:48:04.503 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.504+0000 7fce17248980 1 bluefs mount shared_bdev_used = 0 2024-03-09T12:48:04.504 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.504+0000 7fce17248980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T12:48:04.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.506+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T12:48:04.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.506+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T12:48:04.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.506+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T12:48:04.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.506+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T12:48:04.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.507+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T12:48:04.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.507+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T12:48:04.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.507+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T12:48:04.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.507+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T12:48:04.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.507+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T12:48:04.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.507+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T12:48:04.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.507+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T12:48:04.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.507+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T12:48:04.540 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.540+0000 7fce17248980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T12:48:04.540 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.540+0000 7fce17248980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-03-09T12:48:04.540 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.540+0000 7fce17248980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-03-09T12:48:04.540 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.540+0000 7fce17248980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-09T12:48:04.540 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.540+0000 7fce17248980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-03-09T12:48:04.540 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.540+0000 7fce17248980 1 freelist init 2024-03-09T12:48:04.540 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.540+0000 7fce17248980 1 freelist _read_cfg 2024-03-09T12:48:04.540 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.540+0000 7fce17248980 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-09T12:48:04.541 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.541+0000 7fce17248980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 24 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 8.77383e-07 2024-03-09T12:48:04.541 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.541+0000 7fce17248980 1 bluefs umount 2024-03-09T12:48:04.541 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.541+0000 7fce17248980 1 bdev(0x562361252700 td/osd-mapper/0/block) close 2024-03-09T12:48:04.721 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.722+0000 7fce17248980 1 bdev(0x562361252700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-09T12:48:04.722 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.722+0000 7fce17248980 0 bdev(0x562361252700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-09T12:48:04.722 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.722+0000 7fce17248980 1 bdev(0x562361252700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:04.722 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.722+0000 7fce17248980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-09T12:48:04.722 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.722+0000 7fce17248980 1 bluefs mount 2024-03-09T12:48:04.722 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.722+0000 7fce17248980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T12:48:04.728 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.729+0000 7fce17248980 1 bluefs mount shared_bdev_used = 27459584 2024-03-09T12:48:04.728 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.729+0000 7fce17248980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T12:48:04.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.731+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T12:48:04.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.731+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T12:48:04.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.731+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T12:48:04.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.731+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T12:48:04.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.731+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T12:48:04.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.731+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T12:48:04.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.731+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T12:48:04.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.731+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T12:48:04.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.731+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T12:48:04.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.731+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T12:48:04.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.731+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T12:48:04.732 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:04.731+0000 7fce17248980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T12:48:05.490 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:05.491+0000 7fce17248980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T12:48:05.492 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-09T12:48:05.492 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 0b 05 58 02 00 00 7c 37 a4 82 1a 91 41 cc 97 40 |..X...|7....A..@| 2024-03-09T12:48:05.492 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 5a 8e d6 63 9f ec 00 00 00 00 1d 00 00 00 00 00 |Z..c............| 2024-03-09T12:48:05.492 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-09T12:48:05.492 INFO:tasks.workunit.client.0.smithi162.stdout:* 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-09T12:48:05.493 INFO:tasks.workunit.client.0.smithi162.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-09T12:48:05.494 INFO:tasks.workunit.client.0.smithi162.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout:00000220 72 65 1d 00 00 00 05 00 00 00 a0 03 eb 2d f7 f0 |re...........-..| 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout:00000230 4e fb bc e0 a5 66 79 69 f7 16 00 00 00 00 00 00 |N....fyi........| 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout:00000240 00 00 1d 00 00 00 2c 5a ec 65 de 45 a8 33 01 00 |......,Z.e.E.3..| 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1d 00 00 00 |..............| 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout:0000025e 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:05.495 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout:0000003f 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T12:48:05.496 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:05.497 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:05.498 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-09T12:48:05.499 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:05.500 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-09T12:48:05.500 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:05.500 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:05.492+0000 7fce17248980 1 bluefs umount 2024-03-09T12:48:05.500 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:05.492+0000 7fce17248980 1 bdev(0x562361252700 td/osd-mapper/0/block) close 2024-03-09T12:48:05.727 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:05.728+0000 7fce17248980 1 freelist shutdown 2024-03-09T12:48:05.727 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:05.728+0000 7fce17248980 1 bdev(0x562361252000 td/osd-mapper/0/block) close 2024-03-09T12:48:05.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T12:48:05.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-09T12:48:05.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2024-03-09T12:48:06.005 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.005+0000 7fadeb63e980 1 bdev(0x5652d6e42000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-09T12:48:06.005 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.005+0000 7fadeb63e980 0 bdev(0x5652d6e42000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-09T12:48:06.005 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.005+0000 7fadeb63e980 1 bdev(0x5652d6e42000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:06.005 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.005+0000 7fadeb63e980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-09T12:48:06.005 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.005+0000 7fadeb63e980 1 bdev(0x5652d6e42700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-09T12:48:06.005 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.005+0000 7fadeb63e980 0 bdev(0x5652d6e42700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-09T12:48:06.005 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.005+0000 7fadeb63e980 1 bdev(0x5652d6e42700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:06.005 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.005+0000 7fadeb63e980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-09T12:48:06.005 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.005+0000 7fadeb63e980 1 bluefs mount 2024-03-09T12:48:06.024 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.025+0000 7fadeb63e980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T12:48:06.032 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.033+0000 7fadeb63e980 1 bluefs mount shared_bdev_used = 0 2024-03-09T12:48:06.033 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.033+0000 7fadeb63e980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T12:48:06.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.035+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T12:48:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.035+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T12:48:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.035+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T12:48:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.035+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T12:48:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.035+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T12:48:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.035+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T12:48:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.035+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T12:48:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.036+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T12:48:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.036+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T12:48:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.036+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T12:48:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.036+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T12:48:06.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.036+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T12:48:06.067 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.067+0000 7fadeb63e980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T12:48:06.067 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.067+0000 7fadeb63e980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-03-09T12:48:06.067 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.067+0000 7fadeb63e980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-03-09T12:48:06.067 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.067+0000 7fadeb63e980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-09T12:48:06.067 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.067+0000 7fadeb63e980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-03-09T12:48:06.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.068+0000 7fadeb63e980 1 freelist init 2024-03-09T12:48:06.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.068+0000 7fadeb63e980 1 freelist _read_cfg 2024-03-09T12:48:06.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.068+0000 7fadeb63e980 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-09T12:48:06.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.068+0000 7fadeb63e980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 4.95912e-07 2024-03-09T12:48:06.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.068+0000 7fadeb63e980 1 bluefs umount 2024-03-09T12:48:06.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.068+0000 7fadeb63e980 1 bdev(0x5652d6e42700 td/osd-mapper/2/block) close 2024-03-09T12:48:06.267 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.268+0000 7fadeb63e980 1 bdev(0x5652d6e42700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-09T12:48:06.268 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.268+0000 7fadeb63e980 0 bdev(0x5652d6e42700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-09T12:48:06.268 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.268+0000 7fadeb63e980 1 bdev(0x5652d6e42700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:06.268 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.268+0000 7fadeb63e980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-09T12:48:06.268 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.268+0000 7fadeb63e980 1 bluefs mount 2024-03-09T12:48:06.271 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.272+0000 7fadeb63e980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T12:48:06.277 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.278+0000 7fadeb63e980 1 bluefs mount shared_bdev_used = 27459584 2024-03-09T12:48:06.277 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.278+0000 7fadeb63e980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T12:48:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.280+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T12:48:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.280+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T12:48:06.280 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.280+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T12:48:06.281 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.280+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T12:48:06.281 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.280+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T12:48:06.281 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.280+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T12:48:06.281 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.280+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T12:48:06.281 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.280+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T12:48:06.281 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.280+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T12:48:06.281 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.280+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T12:48:06.281 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.280+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T12:48:06.281 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:06.280+0000 7fadeb63e980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T12:48:07.145 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:07.145+0000 7fadeb63e980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T12:48:07.145 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:07.146+0000 7fadeb63e980 1 bluefs umount 2024-03-09T12:48:07.145 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:07.146+0000 7fadeb63e980 1 bdev(0x5652d6e42700 td/osd-mapper/2/block) close 2024-03-09T12:48:07.381 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:07.382+0000 7fadeb63e980 1 freelist shutdown 2024-03-09T12:48:07.381 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:07.382+0000 7fadeb63e980 1 bdev(0x5652d6e42000 td/osd-mapper/2/block) close 2024-03-09T12:48:07.527 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:07.527 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:07.527 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:07.527 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:07.528 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:07.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T12:48:07.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2024-03-09T12:48:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:grep: /tmp/oo2.dump: No such file or directory 2024-03-09T12:48:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T12:48:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-09T12:48:08.098 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-09T12:48:08.098 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 0b 05 58 02 00 00 7c 37 a4 82 1a 91 41 cc 97 40 |..X...|7....A..@| 2024-03-09T12:48:08.098 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 5a 8e d6 63 9f ec 02 00 00 00 1d 00 00 00 00 00 |Z..c............| 2024-03-09T12:48:08.098 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-09T12:48:08.098 INFO:tasks.workunit.client.0.smithi162.stdout:* 2024-03-09T12:48:08.098 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-09T12:48:08.098 INFO:tasks.workunit.client.0.smithi162.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-09T12:48:08.098 INFO:tasks.workunit.client.0.smithi162.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-09T12:48:08.098 INFO:tasks.workunit.client.0.smithi162.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-09T12:48:08.098 INFO:tasks.workunit.client.0.smithi162.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-09T12:48:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-09T12:48:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-09T12:48:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-09T12:48:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-09T12:48:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-09T12:48:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-09T12:48:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-09T12:48:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-09T12:48:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-09T12:48:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-09T12:48:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-09T12:48:08.099 INFO:tasks.workunit.client.0.smithi162.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-09T12:48:08.101 INFO:tasks.workunit.client.0.smithi162.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-09T12:48:08.101 INFO:tasks.workunit.client.0.smithi162.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-09T12:48:08.101 INFO:tasks.workunit.client.0.smithi162.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-09T12:48:08.101 INFO:tasks.workunit.client.0.smithi162.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-09T12:48:08.101 INFO:tasks.workunit.client.0.smithi162.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-09T12:48:08.101 INFO:tasks.workunit.client.0.smithi162.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-09T12:48:08.101 INFO:tasks.workunit.client.0.smithi162.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-09T12:48:08.101 INFO:tasks.workunit.client.0.smithi162.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout:00000220 72 65 1d 00 00 00 0f 00 00 00 58 e7 70 f2 90 69 |re........X.p..i| 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout:00000230 46 9f 83 f9 86 bc c7 26 50 59 00 00 00 00 00 00 |F......&PY......| 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout:00000240 00 00 1d 00 00 00 70 5a ec 65 d3 19 b6 25 01 00 |......pZ.e...%..| 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1d 00 00 00 |..............| 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout:0000025e 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:08.102 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout:0000003f 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:08.103 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:08.104 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:08.105 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-09T12:48:08.106 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:08.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-09T12:48:08.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-03-09T12:48:08.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-09T12:48:08.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2024-03-09T12:48:10.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2024-03-09T12:48:10.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2024-03-09T12:48:10.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2024-03-09T12:48:10.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T12:48:10.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2024-03-09T12:48:10.121 INFO:tasks.workunit.client.0.smithi162.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2024-03-09T12:48:10.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2024-03-09T12:48:10.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-09T12:48:10.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-09T12:48:10.145 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.145+0000 7f43d104c980 1 bdev(0x55afab346000 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-09T12:48:10.145 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.145+0000 7f43d104c980 0 bdev(0x55afab346000 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-09T12:48:10.146 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.145+0000 7f43d104c980 1 bdev(0x55afab346000 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:10.146 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.145+0000 7f43d104c980 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-09T12:48:10.146 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.145+0000 7f43d104c980 1 bdev(0x55afab346700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-09T12:48:10.146 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.145+0000 7f43d104c980 0 bdev(0x55afab346700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-09T12:48:10.146 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.146+0000 7f43d104c980 1 bdev(0x55afab346700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:10.146 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.146+0000 7f43d104c980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-09T12:48:10.146 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.146+0000 7f43d104c980 1 bluefs mount 2024-03-09T12:48:10.151 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.152+0000 7f43d104c980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T12:48:10.161 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.162+0000 7f43d104c980 1 bluefs mount shared_bdev_used = 0 2024-03-09T12:48:10.161 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.162+0000 7f43d104c980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T12:48:10.164 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.164+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T12:48:10.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.164+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T12:48:10.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.164+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T12:48:10.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.164+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T12:48:10.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.164+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T12:48:10.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.164+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T12:48:10.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.164+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T12:48:10.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.164+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T12:48:10.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.165+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T12:48:10.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.165+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T12:48:10.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.165+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T12:48:10.165 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.165+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T12:48:10.169 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.169+0000 7f43d104c980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T12:48:10.169 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.169+0000 7f43d104c980 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2024-03-09T12:48:10.169 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.169+0000 7f43d104c980 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2024-03-09T12:48:10.169 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.169+0000 7f43d104c980 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-09T12:48:10.169 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.169+0000 7f43d104c980 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2024-03-09T12:48:10.170 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.170+0000 7f43d104c980 1 freelist init 2024-03-09T12:48:10.170 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.170+0000 7f43d104c980 1 freelist _read_cfg 2024-03-09T12:48:10.170 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.170+0000 7f43d104c980 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-09T12:48:10.170 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.170+0000 7f43d104c980 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 24 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 8.77383e-07 2024-03-09T12:48:10.170 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.170+0000 7f43d104c980 1 bluefs umount 2024-03-09T12:48:10.171 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.170+0000 7f43d104c980 1 bdev(0x55afab346700 td/osd-mapper/0/block) close 2024-03-09T12:48:10.407 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.408+0000 7f43d104c980 1 bdev(0x55afab346700 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2024-03-09T12:48:10.408 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.408+0000 7f43d104c980 0 bdev(0x55afab346700 td/osd-mapper/0/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/0/block failed: (22) Invalid argument 2024-03-09T12:48:10.408 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.408+0000 7f43d104c980 1 bdev(0x55afab346700 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:10.408 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.408+0000 7f43d104c980 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2024-03-09T12:48:10.408 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.408+0000 7f43d104c980 1 bluefs mount 2024-03-09T12:48:10.418 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.418+0000 7f43d104c980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T12:48:10.427 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.427+0000 7f43d104c980 1 bluefs mount shared_bdev_used = 9568256 2024-03-09T12:48:10.427 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.427+0000 7f43d104c980 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T12:48:10.429 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.429+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T12:48:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.429+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T12:48:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.429+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T12:48:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.429+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T12:48:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.429+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T12:48:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.430+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T12:48:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.430+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T12:48:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.430+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T12:48:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.430+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T12:48:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.430+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T12:48:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.430+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T12:48:10.430 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.430+0000 7f43d104c980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T12:48:10.752 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.752+0000 7f43d104c980 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T12:48:10.753 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 0b 05 58 02 00 00 7c 37 a4 82 1a 91 41 cc 97 40 |..X...|7....A..@| 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 5a 8e d6 63 9f ec 00 00 00 00 1d 00 00 00 00 00 |Z..c............| 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:* 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-09T12:48:10.754 INFO:tasks.workunit.client.0.smithi162.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-09T12:48:10.755 INFO:tasks.workunit.client.0.smithi162.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout:00000220 72 65 1d 00 00 00 05 00 00 00 a0 03 eb 2d f7 f0 |re...........-..| 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout:00000230 4e fb bc e0 a5 66 79 69 f7 16 00 00 00 00 00 00 |N....fyi........| 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout:00000240 00 00 1d 00 00 00 2c 5a ec 65 de 45 a8 33 01 00 |......,Z.e.E.3..| 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1d 00 00 00 |..............| 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout:0000025e 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:10.756 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout:0000003f 2024-03-09T12:48:10.757 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:10.758 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:10.759 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:10.760 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:10.761 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-09T12:48:10.761 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:10.761 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-09T12:48:10.761 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:10.761 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.753+0000 7f43d104c980 1 bluefs umount 2024-03-09T12:48:10.761 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.753+0000 7f43d104c980 1 bdev(0x55afab346700 td/osd-mapper/0/block) close 2024-03-09T12:48:10.992 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.993+0000 7f43d104c980 1 freelist shutdown 2024-03-09T12:48:10.992 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:10.993+0000 7f43d104c980 1 bdev(0x55afab346000 td/osd-mapper/0/block) close 2024-03-09T12:48:11.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2024-03-09T12:48:11.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-09T12:48:11.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-09T12:48:12.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:12.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T12:48:12.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-09T12:48:12.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-09T12:48:12.206 INFO:tasks.workunit.client.0.smithi162.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:12.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-09T12:48:12.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.07tP7dwiN8_the_val 2024-03-09T12:48:12.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T12:48:12.208 INFO:tasks.workunit.client.0.smithi162.stdout:Value dumped in: /tmp/tmp.07tP7dwiN8_the_val 2024-03-09T12:48:12.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.07tP7dwiN8_the_val' 2024-03-09T12:48:12.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.07tP7dwiN8_the_val 2024-03-09T12:48:12.754 INFO:tasks.workunit.client.0.smithi162.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-09T12:48:13.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T12:48:13.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.07tP7dwiN8_the_val 2024-03-09T12:48:13.273 INFO:tasks.workunit.client.0.smithi162.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-09T12:48:13.273 INFO:tasks.workunit.client.0.smithi162.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-09T12:48:13.273 INFO:tasks.workunit.client.0.smithi162.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-09T12:48:13.273 INFO:tasks.workunit.client.0.smithi162.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-09T12:48:13.274 INFO:tasks.workunit.client.0.smithi162.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-09T12:48:13.274 INFO:tasks.workunit.client.0.smithi162.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-03-09T12:48:13.274 INFO:tasks.workunit.client.0.smithi162.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-09T12:48:13.274 INFO:tasks.workunit.client.0.smithi162.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-09T12:48:13.274 INFO:tasks.workunit.client.0.smithi162.stdout:0000073 2024-03-09T12:48:13.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-09T12:48:13.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:14.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.07tP7dwiN8_the_val 2024-03-09T12:48:15.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T12:48:15.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T12:48:15.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-03-09T12:48:15.364 INFO:tasks.workunit.client.0.smithi162.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2024-03-09T12:48:15.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2024-03-09T12:48:15.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-09T12:48:15.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-09T12:48:15.388 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.388+0000 7f8b09a38980 1 bdev(0x556600b3c000 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-09T12:48:15.388 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.388+0000 7f8b09a38980 0 bdev(0x556600b3c000 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-03-09T12:48:15.388 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.388+0000 7f8b09a38980 1 bdev(0x556600b3c000 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:15.388 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.388+0000 7f8b09a38980 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-09T12:48:15.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.388+0000 7f8b09a38980 1 bdev(0x556600b3c700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-09T12:48:15.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.388+0000 7f8b09a38980 0 bdev(0x556600b3c700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-03-09T12:48:15.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.388+0000 7f8b09a38980 1 bdev(0x556600b3c700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:15.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.388+0000 7f8b09a38980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-03-09T12:48:15.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.388+0000 7f8b09a38980 1 bluefs mount 2024-03-09T12:48:15.392 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.393+0000 7f8b09a38980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T12:48:15.402 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.403+0000 7f8b09a38980 1 bluefs mount shared_bdev_used = 0 2024-03-09T12:48:15.402 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.403+0000 7f8b09a38980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T12:48:15.405 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.405+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T12:48:15.405 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.405+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T12:48:15.406 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.405+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T12:48:15.406 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.405+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T12:48:15.406 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.405+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T12:48:15.406 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.405+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T12:48:15.406 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.405+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T12:48:15.406 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.405+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T12:48:15.406 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.406+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T12:48:15.406 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.406+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T12:48:15.406 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.406+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T12:48:15.406 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.406+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T12:48:15.410 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.410+0000 7f8b09a38980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T12:48:15.410 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.410+0000 7f8b09a38980 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2050 2024-03-09T12:48:15.410 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.410+0000 7f8b09a38980 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2024-03-09T12:48:15.410 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.410+0000 7f8b09a38980 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-09T12:48:15.411 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.410+0000 7f8b09a38980 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2024-03-09T12:48:15.411 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.411+0000 7f8b09a38980 1 freelist init 2024-03-09T12:48:15.411 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.411+0000 7f8b09a38980 1 freelist _read_cfg 2024-03-09T12:48:15.411 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.411+0000 7f8b09a38980 1 bluestore(td/osd-mapper/1) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-09T12:48:15.411 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.411+0000 7f8b09a38980 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 19 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 6.86647e-07 2024-03-09T12:48:15.411 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.411+0000 7f8b09a38980 1 bluefs umount 2024-03-09T12:48:15.412 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.411+0000 7f8b09a38980 1 bdev(0x556600b3c700 td/osd-mapper/1/block) close 2024-03-09T12:48:15.655 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.656+0000 7f8b09a38980 1 bdev(0x556600b3c700 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2024-03-09T12:48:15.656 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.656+0000 7f8b09a38980 0 bdev(0x556600b3c700 td/osd-mapper/1/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/1/block failed: (22) Invalid argument 2024-03-09T12:48:15.656 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.656+0000 7f8b09a38980 1 bdev(0x556600b3c700 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:15.656 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.656+0000 7f8b09a38980 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2024-03-09T12:48:15.656 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.656+0000 7f8b09a38980 1 bluefs mount 2024-03-09T12:48:15.659 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.659+0000 7f8b09a38980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T12:48:15.668 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.668+0000 7f8b09a38980 1 bluefs mount shared_bdev_used = 9568256 2024-03-09T12:48:15.668 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.668+0000 7f8b09a38980 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T12:48:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.670+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T12:48:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.670+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T12:48:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.670+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T12:48:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.670+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T12:48:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.671+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T12:48:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.671+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T12:48:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.671+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T12:48:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.671+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T12:48:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.671+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T12:48:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.671+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T12:48:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.671+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T12:48:15.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.671+0000 7f8b09a38980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T12:48:15.926 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.927+0000 7f8b09a38980 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T12:48:15.927 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-09T12:48:15.928 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 0b 05 58 02 00 00 7c 37 a4 82 1a 91 41 cc 97 40 |..X...|7....A..@| 2024-03-09T12:48:15.928 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 5a 8e d6 63 9f ec 01 00 00 00 1d 00 00 00 00 00 |Z..c............| 2024-03-09T12:48:15.928 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-09T12:48:15.928 INFO:tasks.workunit.client.0.smithi162.stdout:* 2024-03-09T12:48:15.928 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-09T12:48:15.928 INFO:tasks.workunit.client.0.smithi162.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-09T12:48:15.928 INFO:tasks.workunit.client.0.smithi162.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-09T12:48:15.928 INFO:tasks.workunit.client.0.smithi162.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-09T12:48:15.928 INFO:tasks.workunit.client.0.smithi162.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-09T12:48:15.928 INFO:tasks.workunit.client.0.smithi162.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-09T12:48:15.928 INFO:tasks.workunit.client.0.smithi162.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-09T12:48:15.928 INFO:tasks.workunit.client.0.smithi162.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-09T12:48:15.929 INFO:tasks.workunit.client.0.smithi162.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-09T12:48:15.930 INFO:tasks.workunit.client.0.smithi162.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-09T12:48:15.930 INFO:tasks.workunit.client.0.smithi162.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-09T12:48:15.930 INFO:tasks.workunit.client.0.smithi162.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-09T12:48:15.930 INFO:tasks.workunit.client.0.smithi162.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-09T12:48:15.930 INFO:tasks.workunit.client.0.smithi162.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-09T12:48:15.930 INFO:tasks.workunit.client.0.smithi162.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-09T12:48:15.930 INFO:tasks.workunit.client.0.smithi162.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-09T12:48:15.930 INFO:tasks.workunit.client.0.smithi162.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-09T12:48:15.930 INFO:tasks.workunit.client.0.smithi162.stdout:00000220 72 65 1d 00 00 00 0a 00 00 00 ec 75 2f 56 35 98 |re.........u/V5.| 2024-03-09T12:48:15.930 INFO:tasks.workunit.client.0.smithi162.stdout:00000230 42 6c 92 ef 88 ec 74 33 c9 93 00 00 00 00 00 00 |Bl....t3........| 2024-03-09T12:48:15.930 INFO:tasks.workunit.client.0.smithi162.stdout:00000240 00 00 1d 00 00 00 4f 5a ec 65 b5 76 d2 1b 01 00 |......OZ.e.v....| 2024-03-09T12:48:15.930 INFO:tasks.workunit.client.0.smithi162.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1d 00 00 00 |..............| 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout:0000025e 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:15.931 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout:0000003f 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:15.932 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:15.933 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:15.934 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:15.934 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:15.934 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:15.934 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:15.934 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:15.934 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:15.934 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:15.934 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:15.934 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:15.934 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:15.934 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:15.934 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:15.934 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:15.935 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:15.935 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:15.935 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:15.935 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:15.935 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:15.935 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:15.935 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-09T12:48:15.935 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:15.935 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-09T12:48:15.935 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:15.937 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.927+0000 7f8b09a38980 1 bluefs umount 2024-03-09T12:48:15.937 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:15.927+0000 7f8b09a38980 1 bdev(0x556600b3c700 td/osd-mapper/1/block) close 2024-03-09T12:48:16.163 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:16.164+0000 7f8b09a38980 1 freelist shutdown 2024-03-09T12:48:16.163 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:16.164+0000 7f8b09a38980 1 bdev(0x556600b3c000 td/osd-mapper/1/block) close 2024-03-09T12:48:16.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-09T12:48:16.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-09T12:48:16.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-09T12:48:17.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:17.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T12:48:17.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-09T12:48:17.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-09T12:48:17.448 INFO:tasks.workunit.client.0.smithi162.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:17.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-09T12:48:17.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.qmahMp14mS_the_val 2024-03-09T12:48:17.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T12:48:17.450 INFO:tasks.workunit.client.0.smithi162.stdout:Value dumped in: /tmp/tmp.qmahMp14mS_the_val 2024-03-09T12:48:17.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.qmahMp14mS_the_val' 2024-03-09T12:48:17.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.qmahMp14mS_the_val 2024-03-09T12:48:17.937 INFO:tasks.workunit.client.0.smithi162.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-09T12:48:18.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T12:48:18.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.qmahMp14mS_the_val 2024-03-09T12:48:18.243 INFO:tasks.workunit.client.0.smithi162.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-09T12:48:18.243 INFO:tasks.workunit.client.0.smithi162.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-09T12:48:18.243 INFO:tasks.workunit.client.0.smithi162.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-09T12:48:18.243 INFO:tasks.workunit.client.0.smithi162.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-09T12:48:18.243 INFO:tasks.workunit.client.0.smithi162.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-09T12:48:18.243 INFO:tasks.workunit.client.0.smithi162.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-03-09T12:48:18.243 INFO:tasks.workunit.client.0.smithi162.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-09T12:48:18.244 INFO:tasks.workunit.client.0.smithi162.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-09T12:48:18.244 INFO:tasks.workunit.client.0.smithi162.stdout:0000073 2024-03-09T12:48:18.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-09T12:48:18.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.qmahMp14mS_the_val 2024-03-09T12:48:20.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T12:48:20.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T12:48:20.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2024-03-09T12:48:20.327 INFO:tasks.workunit.client.0.smithi162.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2024-03-09T12:48:20.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2024-03-09T12:48:20.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2024-03-09T12:48:20.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-09T12:48:20.353 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.354+0000 7fa66aa18980 1 bdev(0x564f6cd7c000 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-09T12:48:20.353 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.354+0000 7fa66aa18980 0 bdev(0x564f6cd7c000 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-09T12:48:20.353 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.354+0000 7fa66aa18980 1 bdev(0x564f6cd7c000 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:20.353 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.354+0000 7fa66aa18980 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 1073741824 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2024-03-09T12:48:20.353 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.354+0000 7fa66aa18980 1 bdev(0x564f6cd7c700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-09T12:48:20.353 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.354+0000 7fa66aa18980 0 bdev(0x564f6cd7c700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-09T12:48:20.354 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.354+0000 7fa66aa18980 1 bdev(0x564f6cd7c700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:20.354 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.354+0000 7fa66aa18980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-09T12:48:20.354 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.354+0000 7fa66aa18980 1 bluefs mount 2024-03-09T12:48:20.362 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.363+0000 7fa66aa18980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T12:48:20.372 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.372+0000 7fa66aa18980 1 bluefs mount shared_bdev_used = 0 2024-03-09T12:48:20.372 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.373+0000 7fa66aa18980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T12:48:20.375 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.375+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T12:48:20.375 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.375+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T12:48:20.375 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.375+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T12:48:20.375 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.375+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T12:48:20.375 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.375+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T12:48:20.375 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.375+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T12:48:20.376 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.375+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T12:48:20.376 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.375+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T12:48:20.376 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.375+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T12:48:20.376 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.375+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T12:48:20.376 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.375+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T12:48:20.376 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.375+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T12:48:20.380 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.380+0000 7fa66aa18980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T12:48:20.380 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.380+0000 7fa66aa18980 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2050 2024-03-09T12:48:20.380 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.380+0000 7fa66aa18980 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2024-03-09T12:48:20.380 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.380+0000 7fa66aa18980 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2024-03-09T12:48:20.380 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.380+0000 7fa66aa18980 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2024-03-09T12:48:20.380 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.380+0000 7fa66aa18980 1 freelist init 2024-03-09T12:48:20.380 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.380+0000 7fa66aa18980 1 freelist _read_cfg 2024-03-09T12:48:20.380 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.380+0000 7fa66aa18980 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = bitmap, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2024-03-09T12:48:20.381 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.381+0000 7fa66aa18980 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 14 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffbe000, fragmentation 4.95912e-07 2024-03-09T12:48:20.381 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.381+0000 7fa66aa18980 1 bluefs umount 2024-03-09T12:48:20.381 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.381+0000 7fa66aa18980 1 bdev(0x564f6cd7c700 td/osd-mapper/2/block) close 2024-03-09T12:48:20.618 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.619+0000 7fa66aa18980 1 bdev(0x564f6cd7c700 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2024-03-09T12:48:20.618 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.619+0000 7fa66aa18980 0 bdev(0x564f6cd7c700 td/osd-mapper/2/block) ioctl(F_SET_FILE_RW_HINT) on td/osd-mapper/2/block failed: (22) Invalid argument 2024-03-09T12:48:20.619 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.619+0000 7fa66aa18980 1 bdev(0x564f6cd7c700 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) rotational device, discard not supported 2024-03-09T12:48:20.619 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.619+0000 7fa66aa18980 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2024-03-09T12:48:20.619 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.619+0000 7fa66aa18980 1 bluefs mount 2024-03-09T12:48:20.637 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.638+0000 7fa66aa18980 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2024-03-09T12:48:20.664 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.665+0000 7fa66aa18980 1 bluefs mount shared_bdev_used = 9568256 2024-03-09T12:48:20.664 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.665+0000 7fa66aa18980 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2024-03-09T12:48:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.666+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 0, name: default) 2024-03-09T12:48:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.666+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 1, name: m-0) 2024-03-09T12:48:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.666+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 2, name: m-1) 2024-03-09T12:48:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.666+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 3, name: m-2) 2024-03-09T12:48:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.666+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 4, name: p-0) 2024-03-09T12:48:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.666+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 5, name: p-1) 2024-03-09T12:48:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.667+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 6, name: p-2) 2024-03-09T12:48:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.667+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 7, name: O-0) 2024-03-09T12:48:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.667+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 8, name: O-1) 2024-03-09T12:48:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.667+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 9, name: O-2) 2024-03-09T12:48:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.667+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 10, name: L) 2024-03-09T12:48:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.667+0000 7fa66aa18980 2 rocksdb: [db/column_family.cc:578] Failed to register data paths of column family (id: 11, name: P) 2024-03-09T12:48:20.900 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.901+0000 7fa66aa18980 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2024-03-09T12:48:20.901 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2024-03-09T12:48:20.902 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 0b 05 58 02 00 00 7c 37 a4 82 1a 91 41 cc 97 40 |..X...|7....A..@| 2024-03-09T12:48:20.902 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 5a 8e d6 63 9f ec 02 00 00 00 1d 00 00 00 00 00 |Z..c............| 2024-03-09T12:48:20.902 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2024-03-09T12:48:20.902 INFO:tasks.workunit.client.0.smithi162.stdout:* 2024-03-09T12:48:20.902 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2024-03-09T12:48:20.902 INFO:tasks.workunit.client.0.smithi162.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2024-03-09T12:48:20.902 INFO:tasks.workunit.client.0.smithi162.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2024-03-09T12:48:20.902 INFO:tasks.workunit.client.0.smithi162.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2024-03-09T12:48:20.902 INFO:tasks.workunit.client.0.smithi162.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2024-03-09T12:48:20.902 INFO:tasks.workunit.client.0.smithi162.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2024-03-09T12:48:20.902 INFO:tasks.workunit.client.0.smithi162.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2024-03-09T12:48:20.902 INFO:tasks.workunit.client.0.smithi162.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2024-03-09T12:48:20.902 INFO:tasks.workunit.client.0.smithi162.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2024-03-09T12:48:20.903 INFO:tasks.workunit.client.0.smithi162.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2024-03-09T12:48:20.904 INFO:tasks.workunit.client.0.smithi162.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2024-03-09T12:48:20.904 INFO:tasks.workunit.client.0.smithi162.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2024-03-09T12:48:20.904 INFO:tasks.workunit.client.0.smithi162.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2024-03-09T12:48:20.904 INFO:tasks.workunit.client.0.smithi162.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2024-03-09T12:48:20.904 INFO:tasks.workunit.client.0.smithi162.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2024-03-09T12:48:20.904 INFO:tasks.workunit.client.0.smithi162.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2024-03-09T12:48:20.904 INFO:tasks.workunit.client.0.smithi162.stdout:00000220 72 65 1d 00 00 00 0f 00 00 00 58 e7 70 f2 90 69 |re........X.p..i| 2024-03-09T12:48:20.904 INFO:tasks.workunit.client.0.smithi162.stdout:00000230 46 9f 83 f9 86 bc c7 26 50 59 00 00 00 00 00 00 |F......&PY......| 2024-03-09T12:48:20.904 INFO:tasks.workunit.client.0.smithi162.stdout:00000240 00 00 1d 00 00 00 70 5a ec 65 d3 19 b6 25 01 00 |......pZ.e...%..| 2024-03-09T12:48:20.904 INFO:tasks.workunit.client.0.smithi162.stdout:00000250 00 00 01 00 00 00 01 00 00 00 1d 00 00 00 |..............| 2024-03-09T12:48:20.904 INFO:tasks.workunit.client.0.smithi162.stdout:0000025e 2024-03-09T12:48:20.904 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:20.904 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:20.905 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2024-03-09T12:48:20.906 INFO:tasks.workunit.client.0.smithi162.stdout:0000003f 2024-03-09T12:48:20.906 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:20.906 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:20.906 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2024-03-09T12:48:20.906 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2024-03-09T12:48:20.906 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2024-03-09T12:48:20.906 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2024-03-09T12:48:20.906 INFO:tasks.workunit.client.0.smithi162.stdout:00000040 00 00 00 00 00 00 00 |.......| 2024-03-09T12:48:20.906 INFO:tasks.workunit.client.0.smithi162.stdout:00000047 2024-03-09T12:48:20.906 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:20.906 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:20.906 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:20.906 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:20.907 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2024-03-09T12:48:20.908 INFO:tasks.workunit.client.0.smithi162.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2024-03-09T12:48:20.909 INFO:tasks.workunit.client.0.smithi162.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2024-03-09T12:48:20.909 INFO:tasks.workunit.client.0.smithi162.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2024-03-09T12:48:20.909 INFO:tasks.workunit.client.0.smithi162.stdout:0000003b 2024-03-09T12:48:20.909 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:20.909 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2024-03-09T12:48:20.909 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:20.909 INFO:tasks.workunit.client.0.smithi162.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2024-03-09T12:48:20.909 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:20.909 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.901+0000 7fa66aa18980 1 bluefs umount 2024-03-09T12:48:20.910 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:20.901+0000 7fa66aa18980 1 bdev(0x564f6cd7c700 td/osd-mapper/2/block) close 2024-03-09T12:48:21.137 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:21.139+0000 7fa66aa18980 1 freelist shutdown 2024-03-09T12:48:21.137 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:21.139+0000 7fa66aa18980 1 bdev(0x564f6cd7c000 td/osd-mapper/2/block) close 2024-03-09T12:48:21.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2024-03-09T12:48:21.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2024-03-09T12:48:21.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-09T12:48:22.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:22.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T12:48:22.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2024-03-09T12:48:22.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2024-03-09T12:48:22.172 INFO:tasks.workunit.client.0.smithi162.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:22.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2024-03-09T12:48:22.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.L7JbvFzgvM_the_val 2024-03-09T12:48:22.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T12:48:22.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.L7JbvFzgvM_the_val' 2024-03-09T12:48:22.174 INFO:tasks.workunit.client.0.smithi162.stdout:Value dumped in: /tmp/tmp.L7JbvFzgvM_the_val 2024-03-09T12:48:22.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.L7JbvFzgvM_the_val 2024-03-09T12:48:22.727 INFO:tasks.workunit.client.0.smithi162.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2024-03-09T12:48:22.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T12:48:22.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.L7JbvFzgvM_the_val 2024-03-09T12:48:22.974 INFO:tasks.workunit.client.0.smithi162.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2024-03-09T12:48:22.974 INFO:tasks.workunit.client.0.smithi162.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2024-03-09T12:48:22.974 INFO:tasks.workunit.client.0.smithi162.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2024-03-09T12:48:22.974 INFO:tasks.workunit.client.0.smithi162.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2024-03-09T12:48:22.975 INFO:tasks.workunit.client.0.smithi162.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2024-03-09T12:48:22.975 INFO:tasks.workunit.client.0.smithi162.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2024-03-09T12:48:22.975 INFO:tasks.workunit.client.0.smithi162.stdout:0000060 0000 0100 0000 0000 0000 0000 2024-03-09T12:48:22.975 INFO:tasks.workunit.client.0.smithi162.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2024-03-09T12:48:22.975 INFO:tasks.workunit.client.0.smithi162.stdout:0000073 2024-03-09T12:48:22.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-09T12:48:22.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:24.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.L7JbvFzgvM_the_val 2024-03-09T12:48:25.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T12:48:25.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 ' 2024-03-09T12:48:25.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 2024-03-09T12:48:25.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T12:48:25.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T12:48:25.316 INFO:tasks.workunit.client.0.smithi162.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ 2024-03-09T12:48:25.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1/' 2024-03-09T12:48:25.316 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2024-03-09T12:48:25.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2024-03-09T12:48:25.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T12:48:25.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=7c37a482-1a91-41cc-9740-5a8ed6639fec --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T12:48:25.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2024-03-09T12:48:25.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-09T12:48:25.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T12:48:25.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T12:48:25.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T12:48:25.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/0 2024-03-09T12:48:25.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7c37a482-1a91-41cc-9740-5a8ed6639fec --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T12:48:25.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:48:25.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:48:25.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:48:25.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2024-03-09T12:48:25.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2024-03-09T12:48:25.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T12:48:25.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T12:48:25.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-09T12:48:25.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T12:48:25.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:48:25.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:48:25.323 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:48:25.323 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:48:25.323 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28754 2024-03-09T12:48:25.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28754/$cluster-$name.asok' 2024-03-09T12:48:25.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28754/$cluster-$name.asok' 2024-03-09T12:48:25.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:48:25.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-09T12:48:25.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-09T12:48:25.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:48:25.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:48:25.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:48:25.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:48:25.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T12:48:25.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T12:48:25.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/0 2024-03-09T12:48:25.324 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T12:48:25.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T12:48:25.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7c37a482-1a91-41cc-9740-5a8ed6639fec --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T12:48:25.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/0/whoami 2024-03-09T12:48:25.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T12:48:25.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T12:48:25.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T12:48:25.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T12:48:25.361 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:25.362+0000 7f19caf91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:48:25.376 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:25.377+0000 7f19caf91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:48:25.377 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:25.378+0000 7f19caf91640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:48:25.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T12:48:25.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:48:25.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T12:48:25.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:48:25.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:48:25.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:48:25.616 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:48:25.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:48:25.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:48:25.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:48:25.895 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:25.896+0000 7f19caf91640 -1 Falling back to public interface 2024-03-09T12:48:25.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:48:26.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:48:26.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:48:26.900 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:48:26.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:48:26.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:48:26.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:48:27.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:48:28.115 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:28.117+0000 7f19caf91640 -1 osd.0 29 log_to_monitors true 2024-03-09T12:48:28.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:48:28.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:48:28.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:48:28.190 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:48:28.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:48:28.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:48:28.514 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 32 up_thru 17 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/3431304942,v1:127.0.0.1:6803/3431304942] [v2:127.0.0.1:6804/3431304942,v1:127.0.0.1:6805/3431304942] exists,up a003eb2d-f7f0-4efb-bce0-a5667969f716 2024-03-09T12:48:28.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:48:28.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:48:28.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:48:28.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T12:48:28.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=7c37a482-1a91-41cc-9740-5a8ed6639fec --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T12:48:28.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2024-03-09T12:48:28.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-09T12:48:28.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T12:48:28.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T12:48:28.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T12:48:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/1 2024-03-09T12:48:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7c37a482-1a91-41cc-9740-5a8ed6639fec --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T12:48:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:48:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:48:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:48:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2024-03-09T12:48:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2024-03-09T12:48:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T12:48:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T12:48:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-09T12:48:28.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T12:48:28.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:48:28.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28754 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28754/$cluster-$name.asok' 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28754/$cluster-$name.asok' 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T12:48:28.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/1 2024-03-09T12:48:28.522 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T12:48:28.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T12:48:28.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=7c37a482-1a91-41cc-9740-5a8ed6639fec --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T12:48:28.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/1/whoami 2024-03-09T12:48:28.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T12:48:28.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T12:48:28.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T12:48:28.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T12:48:28.553 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:28.555+0000 7f0b430c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:48:28.690 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:28.692+0000 7f0b430c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:48:28.691 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:28.693+0000 7f0b430c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:48:28.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T12:48:28.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:48:28.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T12:48:28.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:48:28.979 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:48:28.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:48:28.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:48:28.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:48:28.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:48:28.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:48:29.206 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:29.208+0000 7f0b430c8640 -1 Falling back to public interface 2024-03-09T12:48:29.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:48:30.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:48:30.308 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:48:30.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:48:30.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:48:30.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:48:30.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:48:30.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:48:31.523 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:31.524+0000 7f0b430c8640 -1 osd.1 29 log_to_monitors true 2024-03-09T12:48:31.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:48:31.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:48:31.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:48:31.602 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:48:31.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:48:31.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:48:31.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:48:32.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:48:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:48:32.892 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:48:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:48:32.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:48:32.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:48:33.184 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 30 last_clean_interval [10,29) [v2:127.0.0.1:6810/673085159,v1:127.0.0.1:6811/673085159] [v2:127.0.0.1:6812/673085159,v1:127.0.0.1:6813/673085159] exists,up ec752f56-3598-426c-92ef-88ec7433c993 2024-03-09T12:48:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:48:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:48:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:48:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T12:48:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=7c37a482-1a91-41cc-9740-5a8ed6639fec --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T12:48:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2024-03-09T12:48:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-mapper 2024-03-09T12:48:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T12:48:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T12:48:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T12:48:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-mapper/2 2024-03-09T12:48:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7c37a482-1a91-41cc-9740-5a8ed6639fec --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2024-03-09T12:48:33.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28754 2024-03-09T12:48:33.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.28754/$cluster-$name.asok' 2024-03-09T12:48:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.28754/$cluster-$name.asok' 2024-03-09T12:48:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:48:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2024-03-09T12:48:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2024-03-09T12:48:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:48:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:48:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:48:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:48:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T12:48:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T12:48:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-mapper/2 2024-03-09T12:48:33.192 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T12:48:33.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T12:48:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=7c37a482-1a91-41cc-9740-5a8ed6639fec --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.28754/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T12:48:33.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-mapper/2/whoami 2024-03-09T12:48:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T12:48:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T12:48:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T12:48:33.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T12:48:33.223 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:33.225+0000 7fbb7d1df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:48:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:33.255+0000 7fbb7d1df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:48:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:33.256+0000 7fbb7d1df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:48:33.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T12:48:33.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:48:33.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T12:48:33.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:48:33.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:48:33.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:48:33.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:48:33.522 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:48:33.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:48:33.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T12:48:33.769 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:33.770+0000 7fbb7d1df640 -1 Falling back to public interface 2024-03-09T12:48:33.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:48:34.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:48:34.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:48:34.815 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:48:34.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:48:34.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:48:34.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T12:48:35.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:48:36.024 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:36.026+0000 7fbb7d1df640 -1 osd.2 29 log_to_monitors true 2024-03-09T12:48:36.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:48:36.109 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:48:36.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:48:36.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:48:36.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:48:36.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T12:48:36.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:48:37.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:48:37.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:48:37.402 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:48:37.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:48:37.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:48:37.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T12:48:37.748 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 39 up_thru 39 down_at 30 last_clean_interval [15,29) [v2:127.0.0.1:6818/458424492,v1:127.0.0.1:6819/458424492] [v2:127.0.0.1:6820/458424492,v1:127.0.0.1:6821/458424492] exists,up 58e770f2-9069-469f-83f9-86bcc7265059 2024-03-09T12:48:37.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:48:37.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:48:37.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:48:37.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2024-03-09T12:48:38.750 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2024-03-09T12:48:38.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2024-03-09T12:48:38.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T12:48:38.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2024-03-09T12:48:38.858 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T12:48:38.858 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "19.0.0-1920-g6b01fe95", 2024-03-09T12:48:38.858 INFO:tasks.workunit.client.0.smithi162.stdout: "release": "squid", 2024-03-09T12:48:38.858 INFO:tasks.workunit.client.0.smithi162.stdout: "release_type": "dev" 2024-03-09T12:48:38.859 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:48:38.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T12:48:38.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2024-03-09T12:48:38.964 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T12:48:38.964 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "19.0.0-1920-g6b01fe95", 2024-03-09T12:48:38.964 INFO:tasks.workunit.client.0.smithi162.stdout: "release": "squid", 2024-03-09T12:48:38.964 INFO:tasks.workunit.client.0.smithi162.stdout: "release_type": "dev" 2024-03-09T12:48:38.964 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:48:38.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2024-03-09T12:48:38.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2024-03-09T12:48:39.070 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T12:48:39.070 INFO:tasks.workunit.client.0.smithi162.stdout: "version": "19.0.0-1920-g6b01fe95", 2024-03-09T12:48:39.071 INFO:tasks.workunit.client.0.smithi162.stdout: "release": "squid", 2024-03-09T12:48:39.071 INFO:tasks.workunit.client.0.smithi162.stdout: "release_type": "dev" 2024-03-09T12:48:39.071 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:48:39.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2024-03-09T12:48:39.107 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T12:48:39.107 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "objxxx", 2024-03-09T12:48:39.107 INFO:tasks.workunit.client.0.smithi162.stdout: "clones": [ 2024-03-09T12:48:39.107 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:48:39.107 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 2, 2024-03-09T12:48:39.107 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [ 2024-03-09T12:48:39.107 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:48:39.107 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 1, 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap01" 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 2, 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap02" 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6, 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: "overlaps": [] 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 3, 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [ 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 3, 2024-03-09T12:48:39.108 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap13" 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6, 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: "overlaps": [] 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 5, 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [ 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 4, 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap24" 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: "id": 5, 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: "name": "snap25" 2024-03-09T12:48:39.109 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-03-09T12:48:39.110 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-03-09T12:48:39.110 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 5, 2024-03-09T12:48:39.110 INFO:tasks.workunit.client.0.smithi162.stdout: "overlaps": [] 2024-03-09T12:48:39.110 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T12:48:39.110 INFO:tasks.workunit.client.0.smithi162.stdout: { 2024-03-09T12:48:39.110 INFO:tasks.workunit.client.0.smithi162.stdout: "id": "head", 2024-03-09T12:48:39.110 INFO:tasks.workunit.client.0.smithi162.stdout: "snapshots": [], 2024-03-09T12:48:39.110 INFO:tasks.workunit.client.0.smithi162.stdout: "size": 6 2024-03-09T12:48:39.110 INFO:tasks.workunit.client.0.smithi162.stdout: } 2024-03-09T12:48:39.110 INFO:tasks.workunit.client.0.smithi162.stdout: ] 2024-03-09T12:48:39.110 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:48:39.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2024-03-09T12:48:39.585 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is unset 2024-03-09T12:48:39.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2024-03-09T12:48:40.079 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is unset 2024-03-09T12:48:40.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2024-03-09T12:48:40.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2024-03-09T12:48:40.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2024-03-09T12:48:40.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-09T12:48:40.632 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T12:48:40.632 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 peering 2024-03-09T12:48:36.615142+0000 27'7 38:58 [1,2,0] 1 [1,2,0] 1 27'7 2024-03-09T12:47:55.082791+0000 27'7 2024-03-09T12:47:55.082791+0000 0 1 no scrub is scheduled 4 0 2024-03-09T12:48:40.632 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:48:37.074208+0000 0'0 41:40 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:48:40.632 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-03-09T12:48:32.790676+0000 0'0 37:17 [0,1] 0 [0,1] 0 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:48:40.633 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 peering 2024-03-09T12:48:36.615108+0000 0'0 38:23 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:48:40.633 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:40.633 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T12:48:40.633 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:48:40.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2024-03-09T12:48:42.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-09T12:48:42.749 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T12:48:42.749 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": true, 2024-03-09T12:48:42.749 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2024-03-09T12:48:42.749 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2024-03-09T12:48:42.749 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:48:42.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2024-03-09T12:48:47.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-09T12:48:48.025 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T12:48:48.025 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-03-09T12:48:37.504431+0000 27'7 43:72 [1,2,0] 1 [1,2,0] 1 27'7 2024-03-09T12:47:55.082791+0000 27'7 2024-03-09T12:47:55.082791+0000 0 1 periodic scrub scheduled @ 2024-03-10T12:47:55.082791+0000 4 0 2024-03-09T12:48:48.025 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:48:37.074208+0000 0'0 43:44 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:48:48.025 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:48:37.504439+0000 0'0 43:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:48:48.026 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:48:37.504360+0000 0'0 43:36 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:48:48.026 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:48.026 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T12:48:48.026 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:48:48.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T12:48:48.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-09T12:48:48.037 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:48:43.488+0000 7f0b2583e640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-03-09T12:48:48.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2024-03-09T12:48:48.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-09T12:48:48.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2024-03-09T12:48:48.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2024-03-09T12:48:48.042 INFO:tasks.workunit.client.0.smithi162.stdout:prev count: 1 2024-03-09T12:48:48.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2024-03-09T12:48:48.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2024-03-09T12:48:48.137 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T12:48:48.138 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": true, 2024-03-09T12:48:48.138 INFO:tasks.workunit.client.0.smithi162.stdout: "must": true, 2024-03-09T12:48:48.138 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "0.000000" 2024-03-09T12:48:48.138 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:48:48.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2024-03-09T12:48:53.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2024-03-09T12:48:53.415 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T12:48:53.416 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 4 0 0 0 0 23 0 0 7 0 7 active+clean 2024-03-09T12:48:48.457636+0000 27'7 43:90 [1,2,0] 1 [1,2,0] 1 27'7 2024-03-09T12:48:48.457551+0000 27'7 2024-03-09T12:48:48.457551+0000 0 1 periodic scrub scheduled @ 2024-03-10T12:48:48.457551+0000 4 0 2024-03-09T12:48:53.416 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:48:37.074208+0000 0'0 43:44 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:48:53.416 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:48:37.504439+0000 0'0 43:29 [2,0,1] 2 [2,0,1] 2 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:48:53.416 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:48:37.504360+0000 0'0 43:36 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-09T12:47:46.415961+0000 0'0 2024-03-09T12:47:46.415961+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:47:46.415961+0000 0 0 2024-03-09T12:48:53.416 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:48:53.416 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T12:48:53.416 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:48:53.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T12:48:53.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-09T12:48:53.428 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:48:43.488+0000 7f0b2583e640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2024-03-09T12:48:53.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2024-03-09T12:48:53.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2024-03-09T12:48:53.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2024-03-09T12:48:53.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2024-03-09T12:48:53.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2024-03-09T12:48:53.431 INFO:tasks.workunit.client.0.smithi162.stdout:current count: 1 2024-03-09T12:48:53.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2024-03-09T12:48:53.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2024-03-09T12:48:53.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:48:53.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:48:53.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:48:53.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:48:53.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:48:53.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:48:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2024-03-09T12:48:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2024-03-09T12:48:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-09T12:48:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2024-03-09T12:48:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-09T12:48:55.275 INFO:tasks.workunit.client.0.smithi162.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:55.275 INFO:tasks.workunit.client.0.smithi162.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2024-03-09T12:48:55.275 INFO:tasks.workunit.client.0.smithi162.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2024-03-09T12:48:55.275 INFO:tasks.workunit.client.0.smithi162.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2024-03-09T12:48:55.275 INFO:tasks.workunit.client.0.smithi162.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:55.275 INFO:tasks.workunit.client.0.smithi162.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2024-03-09T12:48:55.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2024-03-09T12:48:55.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2024-03-09T12:48:55.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2024-03-09T12:48:55.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2024-03-09T12:48:56.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2024-03-09T12:48:56.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2024-03-09T12:48:56.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2024-03-09T12:48:56.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2024-03-09T12:48:56.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-09T12:48:56.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T12:48:56.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-09T12:48:56.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:48:56.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:48:56.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:48:56.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:48:56.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:48:56.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:48:56.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T12:48:56.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T12:48:56.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T12:48:56.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T12:48:56.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T12:48:56.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T12:48:56.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:48:56.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T12:48:56.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T12:48:56.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:48:56.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T12:48:56.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T12:48:56.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-09T12:48:56.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T12:48:56.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:48:56.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28754 2024-03-09T12:48:56.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28754 2024-03-09T12:48:56.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T12:48:56.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T12:48:56.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-09T12:48:56.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-mapper 0 2024-03-09T12:48:56.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-mapper 2024-03-09T12:48:56.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-09T12:48:56.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-mapper KILL 2024-03-09T12:48:56.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:48:56.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:48:56.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:48:56.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:48:56.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:48:56.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:48:56.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T12:48:56.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T12:48:56.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T12:48:56.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T12:48:56.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T12:48:56.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T12:48:56.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:48:56.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T12:48:56.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T12:48:56.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:48:56.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T12:48:56.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-09T12:48:56.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-mapper 2024-03-09T12:48:56.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T12:48:56.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:48:56.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.28754 2024-03-09T12:48:56.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.28754 2024-03-09T12:48:56.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T12:48:56.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T12:48:56.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-09T12:48:56.217 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T12:48:56.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T12:48:56.288 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2024-03-09T12:48:56.288 DEBUG:teuthology.orchestra.run.smithi162:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6b01fe95f3aa686f1951a0c5828907a52930b92a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2024-03-09T12:48:56.351 INFO:tasks.workunit.client.0.smithi162.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-09T12:48:56.356 INFO:tasks.workunit.client.0.smithi162.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-09T12:48:56.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T12:48:56.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T12:48:56.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-09T12:48:56.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-09T12:48:56.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-09T12:48:56.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-09T12:48:56.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-09T12:48:56.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-09T12:48:56.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-recovery-scrub 2024-03-09T12:48:56.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2024-03-09T12:48:56.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2024-03-09T12:48:56.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2024-03-09T12:48:56.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2024-03-09T12:48:56.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2024-03-09T12:48:56.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2024-03-09T12:48:56.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none ' 2024-03-09T12:48:56.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-03-09T12:48:56.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-03-09T12:48:56.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-09T12:48:56.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2024-03-09T12:48:56.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-09T12:48:56.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local 'funcs=TEST_recovery_scrub_1 2024-03-09T12:48:56.360 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_recovery_scrub_2' 2024-03-09T12:48:56.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-03-09T12:48:56.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.44981 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-09T12:48:56.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:48:56.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:48:56.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:48:56.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:48:56.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:48:56.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:48:56.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T12:48:56.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T12:48:56.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T12:48:56.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T12:48:56.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T12:48:56.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T12:48:56.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:48:56.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T12:48:56.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T12:48:56.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:48:56.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T12:48:56.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T12:48:56.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-09T12:48:56.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T12:48:56.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:48:56.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:48:56.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44981 2024-03-09T12:48:56.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T12:48:56.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T12:48:56.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-03-09T12:48:56.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T12:48:56.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:48:56.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:48:56.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.44981 2024-03-09T12:48:56.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T12:48:56.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-09T12:48:56.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-09T12:48:56.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T12:48:56.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-03-09T12:48:56.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-09T12:48:56.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-03-09T12:48:56.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T12:48:56.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T12:48:56.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T12:48:56.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-03-09T12:48:56.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-09T12:48:56.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T12:48:56.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:48:56.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:48:56.812 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:48:56.812 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:48:56.812 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:48:56.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:48:56.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-09T12:48:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T12:48:57.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T12:48:57.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T12:48:57.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T12:48:57.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T12:48:57.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T12:48:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T12:48:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T12:48:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T12:48:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T12:48:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:48:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:48:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44981/ceph-mon.a.asok 2024-03-09T12:48:57.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T12:48:57.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44981/ceph-mon.a.asok config get fsid 2024-03-09T12:48:57.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T12:48:57.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T12:48:57.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T12:48:57.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T12:48:57.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T12:48:57.090 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T12:48:57.090 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T12:48:57.090 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T12:48:57.090 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T12:48:57.090 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:48:57.090 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:48:57.090 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44981/ceph-mon.a.asok 2024-03-09T12:48:57.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T12:48:57.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44981/ceph-mon.a.asok config get mon_host 2024-03-09T12:48:57.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2024-03-09T12:48:57.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-03-09T12:48:57.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T12:48:57.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T12:48:57.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T12:48:57.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-03-09T12:48:57.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T12:48:57.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T12:48:57.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:48:57.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:48:57.982 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:48:57.982 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:48:57.982 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:48:57.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:48:57.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T12:48:57.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T12:48:58.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-03-09T12:48:58.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-03-09T12:48:58.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-03-09T12:48:58.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-03-09T12:48:58.010 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2024-03-09T12:48:58.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2024-03-09T12:48:58.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:48:58.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2024-03-09T12:48:58.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T12:48:58.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:48:58.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T12:48:58.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:48:58.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-03-09T12:48:58.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T12:48:58.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:48:58.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:48:58.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:48:58.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-03-09T12:48:58.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-03-09T12:48:58.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:48:58.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T12:48:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T12:48:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:48:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:48:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:48:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:48:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:48:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-09T12:48:58.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-03-09T12:48:58.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:48:58.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=be7405c8-0155-4322-9c90-93829b28598c 2024-03-09T12:48:58.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 be7405c8-0155-4322-9c90-93829b28598c' 2024-03-09T12:48:58.021 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 be7405c8-0155-4322-9c90-93829b28598c 2024-03-09T12:48:58.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:48:58.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6WuxlZyQCAhAAm+NCbPtwpe01YGGwDi5l4g== 2024-03-09T12:48:58.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6WuxlZyQCAhAAm+NCbPtwpe01YGGwDi5l4g=="}' 2024-03-09T12:48:58.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new be7405c8-0155-4322-9c90-93829b28598c -i td/osd-recovery-scrub/0/new.json 2024-03-09T12:48:58.270 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:48:58.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-03-09T12:48:58.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQC6WuxlZyQCAhAAm+NCbPtwpe01YGGwDi5l4g== --osd-uuid be7405c8-0155-4322-9c90-93829b28598c 2024-03-09T12:48:58.309 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:58.310+0000 7fedf3193640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:48:58.321 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:58.322+0000 7fedf3193640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:48:58.322 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:58.323+0000 7fedf3193640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:48:58.323 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:58.323+0000 7fedf3193640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-09T12:48:58.323 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:58.323+0000 7fedf3193640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-09T12:48:58.323 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:58.323+0000 7fedf3193640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-09T12:48:58.405 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:48:58.405+0000 7fedf3193640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-03-09T12:49:03.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-03-09T12:49:03.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:49:03.906 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T12:49:03.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T12:49:03.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:49:04.262 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T12:49:04.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T12:49:04.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-09T12:49:04.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:49:04.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:49:04.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:49:04.289 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:04.289+0000 7ffbb6e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:04.290 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:04.291+0000 7ffbb6e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:04.291 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:04.292+0000 7ffbb6e0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:04.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T12:49:04.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:49:04.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T12:49:04.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:49:04.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:49:04.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:04.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:49:04.542 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:49:04.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:04.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:49:04.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:49:04.816 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:04.817+0000 7ffbb6e0f640 -1 Falling back to public interface 2024-03-09T12:49:05.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:49:05.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:05.817 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:49:05.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:49:05.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:05.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:49:06.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:49:07.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:49:07.097 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:49:07.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:07.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:49:07.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:07.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:49:07.136 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:07.137+0000 7ffbb6e0f640 -1 osd.0 0 log_to_monitors true 2024-03-09T12:49:07.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:49:08.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:49:08.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:08.403 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:49:08.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:49:08.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:08.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:49:08.691 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1882380213,v1:127.0.0.1:6803/1882380213] [v2:127.0.0.1:6804/1882380213,v1:127.0.0.1:6805/1882380213] exists,up be7405c8-0155-4322-9c90-93829b28598c 2024-03-09T12:49:08.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:49:08.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:49:08.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:49:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:49:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2024-03-09T12:49:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T12:49:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:49:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T12:49:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:49:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-03-09T12:49:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T12:49:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:49:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:49:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:49:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-03-09T12:49:08.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-03-09T12:49:08.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:49:08.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:49:08.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T12:49:08.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:49:08.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:49:08.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:49:08.697 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:49:08.697 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:49:08.697 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:49:08.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:49:08.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:49:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:49:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:49:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:49:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T12:49:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T12:49:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:49:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:49:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:49:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:49:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:49:08.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-09T12:49:08.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-03-09T12:49:08.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:49:08.699 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 36bfb46e-d9a0-468a-ad04-9ff5f828a98d 2024-03-09T12:49:08.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=36bfb46e-d9a0-468a-ad04-9ff5f828a98d 2024-03-09T12:49:08.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 36bfb46e-d9a0-468a-ad04-9ff5f828a98d' 2024-03-09T12:49:08.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:49:08.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDEWuxlV7SAKhAAMea+3TFcKzOM9/oj20R00w== 2024-03-09T12:49:08.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDEWuxlV7SAKhAAMea+3TFcKzOM9/oj20R00w=="}' 2024-03-09T12:49:08.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 36bfb46e-d9a0-468a-ad04-9ff5f828a98d -i td/osd-recovery-scrub/1/new.json 2024-03-09T12:49:09.071 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:49:09.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-03-09T12:49:09.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDEWuxlV7SAKhAAMea+3TFcKzOM9/oj20R00w== --osd-uuid 36bfb46e-d9a0-468a-ad04-9ff5f828a98d 2024-03-09T12:49:09.111 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:09.112+0000 7f225d00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:09.147 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:09.148+0000 7f225d00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:09.148 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:09.149+0000 7f225d00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:09.149 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:09.149+0000 7f225d00f640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-09T12:49:09.149 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:09.149+0000 7f225d00f640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-09T12:49:09.149 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:09.149+0000 7f225d00f640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-09T12:49:09.222 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:09.224+0000 7f225d00f640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-03-09T12:49:15.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-03-09T12:49:15.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:49:15.077 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T12:49:15.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T12:49:15.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:49:15.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T12:49:15.429 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T12:49:15.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-09T12:49:15.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:49:15.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:49:15.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:49:15.455 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:15.457+0000 7fdcf9c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:15.457 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:15.459+0000 7fdcf9c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:15.458 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:15.460+0000 7fdcf9c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:15.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T12:49:15.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:49:15.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T12:49:15.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:49:15.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:49:15.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:15.704 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:49:15.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:49:15.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:15.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:49:15.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:49:15.986 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:15.988+0000 7fdcf9c14640 -1 Falling back to public interface 2024-03-09T12:49:16.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:49:16.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:16.972 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:49:16.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:49:16.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:16.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:49:17.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:49:18.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:49:18.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:18.251 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:49:18.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:49:18.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:18.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:49:18.428 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:18.429+0000 7fdcf9c14640 -1 osd.1 0 log_to_monitors true 2024-03-09T12:49:18.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:49:19.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:49:19.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:19.533 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:49:19.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:49:19.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:19.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:49:19.819 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2900598077,v1:127.0.0.1:6811/2900598077] [v2:127.0.0.1:6812/2900598077,v1:127.0.0.1:6813/2900598077] exists,up 36bfb46e-d9a0-468a-ad04-9ff5f828a98d 2024-03-09T12:49:19.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:49:19.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:49:19.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:49:19.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:49:19.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2024-03-09T12:49:19.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T12:49:19.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:49:19.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T12:49:19.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:49:19.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:49:19.825 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:49:19.826 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:49:19.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:49:19.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:49:19.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:49:19.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:49:19.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:49:19.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T12:49:19.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T12:49:19.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:49:19.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:49:19.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:49:19.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:49:19.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:49:19.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-09T12:49:19.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-03-09T12:49:19.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:49:19.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ba205b65-fddd-4c7f-8c22-331e16afb60b 2024-03-09T12:49:19.827 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 ba205b65-fddd-4c7f-8c22-331e16afb60b 2024-03-09T12:49:19.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ba205b65-fddd-4c7f-8c22-331e16afb60b' 2024-03-09T12:49:19.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:49:19.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPWuxlZA0vMhAAUKquZ9cLyGQzyN+77179yA== 2024-03-09T12:49:19.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPWuxlZA0vMhAAUKquZ9cLyGQzyN+77179yA=="}' 2024-03-09T12:49:19.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ba205b65-fddd-4c7f-8c22-331e16afb60b -i td/osd-recovery-scrub/2/new.json 2024-03-09T12:49:20.229 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:49:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-03-09T12:49:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDPWuxlZA0vMhAAUKquZ9cLyGQzyN+77179yA== --osd-uuid ba205b65-fddd-4c7f-8c22-331e16afb60b 2024-03-09T12:49:20.273 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:20.275+0000 7f5f255bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:20.314 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:20.316+0000 7f5f255bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:20.315 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:20.317+0000 7f5f255bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:20.315 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:20.317+0000 7f5f255bd640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-09T12:49:20.315 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:20.317+0000 7f5f255bd640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-09T12:49:20.315 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:20.317+0000 7f5f255bd640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-09T12:49:20.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:20.390+0000 7f5f255bd640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-03-09T12:49:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-03-09T12:49:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:49:26.071 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T12:49:26.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T12:49:26.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:49:26.429 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T12:49:26.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T12:49:26.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-09T12:49:26.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:49:26.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:49:26.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:49:26.455 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:26.457+0000 7fc189260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:26.457 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:26.459+0000 7fc189260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:26.458 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:26.460+0000 7fc189260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:26.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T12:49:26.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:49:26.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T12:49:26.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:49:26.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:49:26.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:26.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:49:26.705 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:49:26.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:26.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T12:49:26.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:49:26.976 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:26.978+0000 7fc189260640 -1 Falling back to public interface 2024-03-09T12:49:27.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:49:27.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:27.977 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:49:27.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:49:27.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:27.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T12:49:28.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:49:29.260 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:29.262+0000 7fc189260640 -1 osd.2 0 log_to_monitors true 2024-03-09T12:49:29.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:49:29.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:29.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:49:29.261 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:49:29.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:29.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T12:49:29.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:49:30.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:49:30.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:30.562 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:49:30.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:49:30.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:30.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T12:49:30.880 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1600274153,v1:127.0.0.1:6819/1600274153] [v2:127.0.0.1:6820/1600274153,v1:127.0.0.1:6821/1600274153] exists,up ba205b65-fddd-4c7f-8c22-331e16afb60b 2024-03-09T12:49:30.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:49:30.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:49:30.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:49:30.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:49:30.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2024-03-09T12:49:30.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T12:49:30.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:49:30.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-09T12:49:30.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:49:30.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-03-09T12:49:30.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T12:49:30.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:49:30.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:49:30.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:49:30.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:49:30.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:49:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:49:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T12:49:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T12:49:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:49:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:49:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:49:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:49:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:49:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2024-03-09T12:49:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-03-09T12:49:30.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:49:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=078dad5c-c47b-4fc6-b458-140fce14d2d0 2024-03-09T12:49:30.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 078dad5c-c47b-4fc6-b458-140fce14d2d0' 2024-03-09T12:49:30.888 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 078dad5c-c47b-4fc6-b458-140fce14d2d0 2024-03-09T12:49:30.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:49:30.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDaWuxlxznVNRAA8ZGrum2ymHl7/RZ/90SFbA== 2024-03-09T12:49:30.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDaWuxlxznVNRAA8ZGrum2ymHl7/RZ/90SFbA=="}' 2024-03-09T12:49:30.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 078dad5c-c47b-4fc6-b458-140fce14d2d0 -i td/osd-recovery-scrub/3/new.json 2024-03-09T12:49:31.248 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:49:31.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-03-09T12:49:31.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQDaWuxlxznVNRAA8ZGrum2ymHl7/RZ/90SFbA== --osd-uuid 078dad5c-c47b-4fc6-b458-140fce14d2d0 2024-03-09T12:49:31.288 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:31.289+0000 7fa0d7ab3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:31.349 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:31.351+0000 7fa0d7ab3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:31.350 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:31.352+0000 7fa0d7ab3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:31.350 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:31.352+0000 7fa0d7ab3640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-09T12:49:31.350 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:31.352+0000 7fa0d7ab3640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-09T12:49:31.351 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:31.352+0000 7fa0d7ab3640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-09T12:49:31.433 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:31.434+0000 7fa0d7ab3640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-03-09T12:49:37.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-03-09T12:49:37.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:49:37.199 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-03-09T12:49:37.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-09T12:49:37.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:49:37.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-09T12:49:37.550 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-03-09T12:49:37.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2024-03-09T12:49:37.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:49:37.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:49:37.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:49:37.576 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:37.577+0000 7f4ef4a48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:37.578 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:37.579+0000 7f4ef4a48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:37.579 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:37.580+0000 7f4ef4a48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:49:37.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-09T12:49:37.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:49:37.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T12:49:37.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:49:37.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:49:37.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:37.840 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:49:37.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:49:37.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:37.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T12:49:38.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:49:38.124 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:38.125+0000 7f4ef4a48640 -1 Falling back to public interface 2024-03-09T12:49:39.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:49:39.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:39.116 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:49:39.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:49:39.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:39.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T12:49:39.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:49:40.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:49:40.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:40.400 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:49:40.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:49:40.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:40.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T12:49:40.554 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:49:40.554+0000 7f4ef4a48640 -1 osd.3 0 log_to_monitors true 2024-03-09T12:49:40.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:49:41.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:49:41.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:49:41.755 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:49:41.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:49:41.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:49:41.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T12:49:42.039 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1250501554,v1:127.0.0.1:6827/1250501554] [v2:127.0.0.1:6828/1250501554,v1:127.0.0.1:6829/1250501554] exists,up 078dad5c-c47b-4fc6-b458-140fce14d2d0 2024-03-09T12:49:42.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:49:42.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:49:42.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:49:42.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2024-03-09T12:49:42.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-03-09T12:49:42.458 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'test' created 2024-03-09T12:49:42.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T12:49:43.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2024-03-09T12:49:43.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T12:49:43.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T12:49:43.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T12:49:43.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T12:49:43.472 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T12:49:43.472 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T12:49:43.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T12:49:43.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T12:49:43.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T12:49:43.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T12:49:43.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T12:49:43.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T12:49:43.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T12:49:43.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T12:49:43.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T12:49:43.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T12:49:43.862 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T12:49:43.862 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T12:49:43.862 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-03-09T12:49:43.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T12:49:43.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:49:43.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T12:49:43.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836489 2024-03-09T12:49:43.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836489 2024-03-09T12:49:43.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489' 2024-03-09T12:49:43.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:49:43.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T12:49:44.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-03-09T12:49:44.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-03-09T12:49:44.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966' 2024-03-09T12:49:44.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:49:44.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T12:49:44.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-03-09T12:49:44.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-03-09T12:49:44.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509445' 2024-03-09T12:49:44.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:49:44.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T12:49:44.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345922 2024-03-09T12:49:44.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345922 2024-03-09T12:49:44.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836489 1-42949672966 2-64424509445 3-85899345922' 2024-03-09T12:49:44.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:49:44.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836489 2024-03-09T12:49:44.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:49:44.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T12:49:44.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836489 2024-03-09T12:49:44.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:49:44.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836489 2024-03-09T12:49:44.309 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836489 2024-03-09T12:49:44.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-03-09T12:49:44.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:49:44.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2024-03-09T12:49:44.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:49:45.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T12:49:45.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:49:45.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836488 -lt 21474836489 2024-03-09T12:49:45.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:49:46.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T12:49:46.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:49:47.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836489 2024-03-09T12:49:47.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:49:47.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-03-09T12:49:47.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:49:47.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T12:49:47.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-03-09T12:49:47.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:49:47.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-03-09T12:49:47.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-03-09T12:49:47.155 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672966 2024-03-09T12:49:47.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T12:49:47.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672966 2024-03-09T12:49:47.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:49:47.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-03-09T12:49:47.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:49:47.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T12:49:47.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-03-09T12:49:47.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:49:47.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-03-09T12:49:47.434 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509445 2024-03-09T12:49:47.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-03-09T12:49:47.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T12:49:47.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509445 2024-03-09T12:49:47.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:49:47.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345922 2024-03-09T12:49:47.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:49:47.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T12:49:47.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345922 2024-03-09T12:49:47.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:49:47.709 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345922 2024-03-09T12:49:47.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345922 2024-03-09T12:49:47.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345922' 2024-03-09T12:49:47.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T12:49:47.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345922 -lt 85899345922 2024-03-09T12:49:47.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T12:49:47.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:49:47.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:49:48.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T12:49:48.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T12:49:48.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T12:49:48.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T12:49:48.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T12:49:48.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T12:49:48.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T12:49:48.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T12:49:48.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T12:49:48.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T12:49:48.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:49:48.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:49:48.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T12:49:48.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T12:49:48.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T12:49:48.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2024-03-09T12:49:48.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2024-03-09T12:49:48.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2024-03-09T12:49:49.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2024-03-09T12:49:49.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2024-03-09T12:49:49.490 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T12:49:49.490 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:49:43.413136+0000 0'0 24:11 [1] 1 [1] 1 0'0 2024-03-09T12:49:42.408072+0000 0'0 2024-03-09T12:49:42.408072+0000 0 0 periodic scrub scheduled @ 2024-03-10T22:12:32.319044+0000 0 0 2024-03-09T12:49:49.491 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:49:49.491 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T12:49:49.491 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:49:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.44981 bs=1M count=50 2024-03-09T12:49:49.642 INFO:tasks.workunit.client.0.smithi162.stderr:50+0 records in 2024-03-09T12:49:49.642 INFO:tasks.workunit.client.0.smithi162.stderr:50+0 records out 2024-03-09T12:49:49.642 INFO:tasks.workunit.client.0.smithi162.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.139947 s, 375 MB/s 2024-03-09T12:49:49.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2024-03-09T12:49:49.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:49:49.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.44981 2024-03-09T12:49:51.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:49:51.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.44981 2024-03-09T12:49:52.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:49:52.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.44981 2024-03-09T12:49:53.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:49:53.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.44981 2024-03-09T12:49:55.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:49:55.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.44981 2024-03-09T12:49:56.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:49:56.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.44981 2024-03-09T12:49:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:49:58.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.44981 2024-03-09T12:49:59.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:49:59.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.44981 2024-03-09T12:50:01.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:01.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.44981 2024-03-09T12:50:02.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:02.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.44981 2024-03-09T12:50:04.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:04.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.44981 2024-03-09T12:50:05.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:05.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.44981 2024-03-09T12:50:07.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:07.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.44981 2024-03-09T12:50:08.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:08.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.44981 2024-03-09T12:50:10.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:10.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.44981 2024-03-09T12:50:11.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:11.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.44981 2024-03-09T12:50:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.44981 2024-03-09T12:50:14.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:14.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.44981 2024-03-09T12:50:16.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:16.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.44981 2024-03-09T12:50:17.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:17.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.44981 2024-03-09T12:50:19.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:19.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.44981 2024-03-09T12:50:20.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:20.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.44981 2024-03-09T12:50:22.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:22.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.44981 2024-03-09T12:50:23.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:23.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.44981 2024-03-09T12:50:25.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:25.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.44981 2024-03-09T12:50:26.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:26.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.44981 2024-03-09T12:50:28.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:28.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.44981 2024-03-09T12:50:29.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:29.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.44981 2024-03-09T12:50:31.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:31.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.44981 2024-03-09T12:50:32.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:32.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.44981 2024-03-09T12:50:33.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:33.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.44981 2024-03-09T12:50:35.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:35.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.44981 2024-03-09T12:50:36.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:36.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.44981 2024-03-09T12:50:38.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:38.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.44981 2024-03-09T12:50:39.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:39.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.44981 2024-03-09T12:50:41.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:41.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.44981 2024-03-09T12:50:42.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:42.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.44981 2024-03-09T12:50:44.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:44.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.44981 2024-03-09T12:50:45.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:45.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.44981 2024-03-09T12:50:47.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:47.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.44981 2024-03-09T12:50:48.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:48.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.44981 2024-03-09T12:50:50.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:50.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.44981 2024-03-09T12:50:51.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:51.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.44981 2024-03-09T12:50:52.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:52.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.44981 2024-03-09T12:50:54.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:54.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.44981 2024-03-09T12:50:55.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:55.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.44981 2024-03-09T12:50:57.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:57.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.44981 2024-03-09T12:50:58.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:50:58.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.44981 2024-03-09T12:51:00.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:00.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.44981 2024-03-09T12:51:01.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:01.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.44981 2024-03-09T12:51:03.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:03.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.44981 2024-03-09T12:51:04.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:04.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.44981 2024-03-09T12:51:06.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:06.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.44981 2024-03-09T12:51:07.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:07.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.44981 2024-03-09T12:51:09.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:09.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.44981 2024-03-09T12:51:10.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:10.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.44981 2024-03-09T12:51:12.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:12.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.44981 2024-03-09T12:51:13.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:13.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.44981 2024-03-09T12:51:14.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:14.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.44981 2024-03-09T12:51:16.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:16.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.44981 2024-03-09T12:51:17.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:17.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.44981 2024-03-09T12:51:19.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:19.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.44981 2024-03-09T12:51:20.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:20.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.44981 2024-03-09T12:51:22.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:22.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.44981 2024-03-09T12:51:23.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:23.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.44981 2024-03-09T12:51:25.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:25.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.44981 2024-03-09T12:51:26.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:26.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.44981 2024-03-09T12:51:28.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:28.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.44981 2024-03-09T12:51:29.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:29.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.44981 2024-03-09T12:51:31.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:31.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.44981 2024-03-09T12:51:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:32.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.44981 2024-03-09T12:51:34.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:34.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.44981 2024-03-09T12:51:35.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:35.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.44981 2024-03-09T12:51:37.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:37.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.44981 2024-03-09T12:51:38.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:38.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.44981 2024-03-09T12:51:40.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:40.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.44981 2024-03-09T12:51:41.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:41.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.44981 2024-03-09T12:51:43.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:43.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.44981 2024-03-09T12:51:44.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:44.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.44981 2024-03-09T12:51:46.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:46.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.44981 2024-03-09T12:51:47.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:47.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.44981 2024-03-09T12:51:49.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:49.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.44981 2024-03-09T12:51:50.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:50.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.44981 2024-03-09T12:51:52.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:52.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.44981 2024-03-09T12:51:53.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:53.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.44981 2024-03-09T12:51:55.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:55.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.44981 2024-03-09T12:51:56.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:56.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.44981 2024-03-09T12:51:58.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:58.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.44981 2024-03-09T12:51:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:51:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.44981 2024-03-09T12:52:01.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:52:01.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.44981 2024-03-09T12:52:02.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:52:02.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.44981 2024-03-09T12:52:04.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:52:04.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.44981 2024-03-09T12:52:05.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:52:05.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.44981 2024-03-09T12:52:07.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:52:07.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.44981 2024-03-09T12:52:08.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:52:08.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.44981 2024-03-09T12:52:09.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:52:09.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.44981 2024-03-09T12:52:11.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:52:11.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.44981 2024-03-09T12:52:13.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:52:13.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.44981 2024-03-09T12:52:14.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:52:14.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.44981 2024-03-09T12:52:16.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2024-03-09T12:52:16.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.44981 2024-03-09T12:52:17.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.44981 2024-03-09T12:52:17.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2024-03-09T12:52:18.021 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 1 size to 4 2024-03-09T12:52:18.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2024-03-09T12:52:18.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2024-03-09T12:52:18.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-09T12:52:18.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-09T12:52:18.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-09T12:52:18.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-09T12:52:18.354 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:52:18.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-09T12:52:20.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2024-03-09T12:52:20.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2024-03-09T12:52:20.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2024-03-09T12:52:20.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-09T12:52:20.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-09T12:52:20.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-09T12:52:20.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-09T12:52:20.688 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:52:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-09T12:52:22.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2024-03-09T12:52:22.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2024-03-09T12:52:22.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2024-03-09T12:52:22.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-09T12:52:22.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-09T12:52:22.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-09T12:52:22.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-09T12:52:22.976 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:52:22.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-09T12:52:24.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 2 -eq 10 2024-03-09T12:52:24.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 2 + 1 2024-03-09T12:52:24.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=3 2024-03-09T12:52:24.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-09T12:52:24.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-09T12:52:24.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-09T12:52:24.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-09T12:52:25.266 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:52:25.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2024-03-09T12:52:27.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 3 -eq 10 2024-03-09T12:52:27.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 3 + 1 2024-03-09T12:52:27.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=4 2024-03-09T12:52:27.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2024-03-09T12:52:27.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2024-03-09T12:52:27.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2024-03-09T12:52:27.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2024-03-09T12:52:27.545 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:52:27.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2024-03-09T12:52:27.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2024-03-09T12:52:27.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2024-03-09T12:52:27.819 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T12:52:27.820 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 100 0 300 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2024-03-09T12:52:21.780403+0000 24'1300 28:1324 [1,0,2,3] 1 [1,0] 1 0'0 2024-03-09T12:49:42.408072+0000 0'0 2024-03-09T12:49:42.408072+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:52:27.820 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:52:27.820 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T12:52:27.820 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:52:27.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2024-03-09T12:52:37.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2024-03-09T12:52:37.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:52:37.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:52:37.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:52:37.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:52:37.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:52:43.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:52:43.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2024-03-09T12:52:43.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress. Only high priority scrubs allowed.' 2024-03-09T12:52:43.148 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2024-03-09T12:52:43.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2024-03-09T12:52:43.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:52:43.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-03-09T12:52:43.153 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:21.872+0000 7ffbaf78a640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.153 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:23.885+0000 7ffbaf78a640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.153 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:24.901+0000 7ffbaf78a640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.153 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:26.817+0000 7ffbaf78a640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.153 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:30.754+0000 7ffbaf78a640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.153 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:34.791+0000 7ffbaf78a640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.154 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:36.803+0000 7ffbaf78a640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.154 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:37.758+0000 7ffbaf78a640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.154 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:39.769+0000 7ffbaf78a640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:52:43.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-03-09T12:52:43.166 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:22.282+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.166 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:23.233+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.166 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:25.247+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.166 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:26.259+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.167 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:27.248+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.167 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:28.248+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.167 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:30.232+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.167 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:33.256+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.167 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:35.300+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.167 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:36.320+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.167 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:39.305+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.167 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:40.298+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.167 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:41.300+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.167 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:42.309+0000 7fdcf256e640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:52:43.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-03-09T12:52:43.169 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:22.203+0000 7fc182203640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.169 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:28.227+0000 7fc182203640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.169 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:29.247+0000 7fc182203640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.169 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:32.247+0000 7fc182203640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.169 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:34.189+0000 7fc182203640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.169 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:35.179+0000 7fc182203640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.169 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:41.194+0000 7fc182203640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:52:43.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-03-09T12:52:43.172 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:27.653+0000 7f4eed9eb640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.172 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:28.700+0000 7f4eed9eb640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.172 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:31.600+0000 7f4eed9eb640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.172 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:35.599+0000 7f4eed9eb640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.172 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:38.596+0000 7f4eed9eb640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.172 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:52:42.546+0000 7f4eed9eb640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Only high priority scrubs allowed. 2024-03-09T12:52:43.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2024-03-09T12:52:43.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2024-03-09T12:52:43.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2024-03-09T12:52:43.173 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2024-03-09T12:52:43.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2024-03-09T12:52:43.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:52:43.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.0.log 2024-03-09T12:52:43.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-09T12:52:43.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2024-03-09T12:52:43.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2024-03-09T12:52:43.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:52:43.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.1.log 2024-03-09T12:52:43.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-09T12:52:43.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2024-03-09T12:52:43.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2024-03-09T12:52:43.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:52:43.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.2.log 2024-03-09T12:52:43.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-09T12:52:43.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2024-03-09T12:52:43.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2024-03-09T12:52:43.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:52:43.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress. Only high priority scrubs allowed.' td/osd-recovery-scrub/osd.3.log 2024-03-09T12:52:43.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2024-03-09T12:52:43.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2024-03-09T12:52:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2024-03-09T12:52:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2024-03-09T12:52:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2024-03-09T12:52:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2024-03-09T12:52:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-09T12:52:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T12:52:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-09T12:52:43.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:52:43.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:52:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:52:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:52:43.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:52:43.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:52:43.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T12:52:43.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T12:52:43.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T12:52:43.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T12:52:43.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T12:52:43.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T12:52:43.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:52:43.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T12:52:43.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T12:52:43.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:52:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T12:52:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T12:52:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-09T12:52:43.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T12:52:43.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:52:43.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:52:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44981 2024-03-09T12:52:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T12:52:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T12:52:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2024-03-09T12:52:43.235 INFO:tasks.workunit.client.0.smithi162.stdout:TEST PASSED 2024-03-09T12:52:43.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2024-03-09T12:52:43.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2024-03-09T12:52:43.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2024-03-09T12:52:43.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_2 td/osd-recovery-scrub 2024-03-09T12:52:43.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:234: TEST_recovery_scrub_2: local dir=td/osd-recovery-scrub 2024-03-09T12:52:43.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:235: TEST_recovery_scrub_2: local poolname=test 2024-03-09T12:52:43.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:237: TEST_recovery_scrub_2: TESTDATA=testdata.44981 2024-03-09T12:52:43.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:238: TEST_recovery_scrub_2: OSDS=8 2024-03-09T12:52:43.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:239: TEST_recovery_scrub_2: PGS=32 2024-03-09T12:52:43.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:240: TEST_recovery_scrub_2: OBJECTS=40 2024-03-09T12:52:43.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:242: TEST_recovery_scrub_2: setup td/osd-recovery-scrub 2024-03-09T12:52:43.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-scrub 2024-03-09T12:52:43.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-scrub 2024-03-09T12:52:43.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-09T12:52:43.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T12:52:43.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-09T12:52:43.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:52:43.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:52:43.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:52:43.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:52:43.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:52:43.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:52:43.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T12:52:43.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T12:52:43.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T12:52:43.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T12:52:43.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T12:52:43.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T12:52:43.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:52:43.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T12:52:43.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T12:52:43.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:52:43.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T12:52:43.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T12:52:43.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-09T12:52:43.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T12:52:43.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:52:43.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:52:43.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44981 2024-03-09T12:52:43.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T12:52:43.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T12:52:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-scrub 2024-03-09T12:52:43.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T12:52:43.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:52:43.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:52:43.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.44981 2024-03-09T12:52:43.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T12:52:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T12:52:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T12:52:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2024-03-09T12:52:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:243: TEST_recovery_scrub_2: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-09T12:52:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-scrub 2024-03-09T12:52:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T12:52:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T12:52:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T12:52:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-scrub/a 2024-03-09T12:52:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-09T12:52:43.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T12:52:43.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:52:43.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:52:43.978 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:52:43.978 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:52:43.978 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:52:43.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:52:43.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-03-09T12:52:44.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T12:52:44.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T12:52:44.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T12:52:44.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T12:52:44.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T12:52:44.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T12:52:44.182 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T12:52:44.183 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T12:52:44.183 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T12:52:44.183 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T12:52:44.184 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:52:44.184 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:52:44.184 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44981/ceph-mon.a.asok 2024-03-09T12:52:44.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T12:52:44.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44981/ceph-mon.a.asok config get fsid 2024-03-09T12:52:44.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T12:52:44.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T12:52:44.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T12:52:44.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T12:52:44.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T12:52:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T12:52:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T12:52:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T12:52:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T12:52:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:52:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:52:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.44981/ceph-mon.a.asok 2024-03-09T12:52:44.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T12:52:44.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.44981/ceph-mon.a.asok config get mon_host 2024-03-09T12:52:44.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:244: TEST_recovery_scrub_2: run_mgr td/osd-recovery-scrub x 2024-03-09T12:52:44.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-scrub 2024-03-09T12:52:44.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T12:52:44.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T12:52:44.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T12:52:44.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-scrub/x 2024-03-09T12:52:44.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T12:52:45.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T12:52:45.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:52:45.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:52:45.157 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:52:45.157 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:52:45.157 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:52:45.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:52:45.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T12:52:45.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T12:52:45.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:245: TEST_recovery_scrub_2: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2024-03-09T12:52:45.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:246: TEST_recovery_scrub_2: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2024-03-09T12:52:45.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:247: TEST_recovery_scrub_2: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2024-03-09T12:52:45.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:248: TEST_recovery_scrub_2: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2024-03-09T12:52:45.186 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: expr 8 - 1 2024-03-09T12:52:45.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: seq 0 7 2024-03-09T12:52:45.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:52:45.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:52:45.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T12:52:45.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:52:45.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T12:52:45.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:52:45.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/0 2024-03-09T12:52:45.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T12:52:45.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:52:45.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:52:45.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:52:45.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2024-03-09T12:52:45.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2024-03-09T12:52:45.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:52:45.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:52:45.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T12:52:45.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:52:45.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:52:45.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:52:45.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:52:45.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:52:45.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T12:52:45.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/0 2024-03-09T12:52:45.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:52:45.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2c49bdea-4ffe-4e21-bfca-ce5ae9457936 2024-03-09T12:52:45.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2c49bdea-4ffe-4e21-bfca-ce5ae9457936' 2024-03-09T12:52:45.194 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 2c49bdea-4ffe-4e21-bfca-ce5ae9457936 2024-03-09T12:52:45.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:52:45.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdW+xl2FeODBAAtlUB9Ylnxvp8jd+XjtDOxQ== 2024-03-09T12:52:45.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdW+xl2FeODBAAtlUB9Ylnxvp8jd+XjtDOxQ=="}' 2024-03-09T12:52:45.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2c49bdea-4ffe-4e21-bfca-ce5ae9457936 -i td/osd-recovery-scrub/0/new.json 2024-03-09T12:52:45.438 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:52:45.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/0/new.json 2024-03-09T12:52:45.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCdW+xl2FeODBAAtlUB9Ylnxvp8jd+XjtDOxQ== --osd-uuid 2c49bdea-4ffe-4e21-bfca-ce5ae9457936 2024-03-09T12:52:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:45.482+0000 7f66c4d6b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:52:45.488 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:45.490+0000 7f66c4d6b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:52:45.489 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:45.491+0000 7f66c4d6b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:52:45.490 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:45.491+0000 7f66c4d6b640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-09T12:52:45.490 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:45.491+0000 7f66c4d6b640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-09T12:52:45.490 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:45.491+0000 7f66c4d6b640 -1 bluestore(td/osd-recovery-scrub/0/block) _read_bdev_label failed to open td/osd-recovery-scrub/0/block: (2) No such file or directory 2024-03-09T12:52:45.581 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:45.582+0000 7f66c4d6b640 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2024-03-09T12:52:50.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2024-03-09T12:52:50.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:52:50.980 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T12:52:50.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T12:52:50.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:52:51.342 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T12:52:51.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T12:52:51.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:52:51.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:52:51.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:52:51.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:52:51.372 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:51.374+0000 7fc3fbd49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:52:51.390 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:51.391+0000 7fc3fbd49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:52:51.391 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:51.392+0000 7fc3fbd49640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:52:51.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T12:52:51.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:52:51.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T12:52:51.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:52:51.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:52:51.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:52:51.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:52:51.617 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:52:51.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:52:51.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:52:51.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:52:51.910 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:51.912+0000 7fc3fbd49640 -1 Falling back to public interface 2024-03-09T12:52:52.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:52:52.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:52:52.889 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:52:52.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:52:52.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:52:52.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:52:53.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:52:54.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:52:54.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:52:54.169 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:52:54.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:52:54.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:52:54.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:52:54.249 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:54.250+0000 7fc3fbd49640 -1 osd.0 0 log_to_monitors true 2024-03-09T12:52:54.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:52:55.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:52:55.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:52:55.493 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:52:55.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:52:55.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:52:55.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:52:55.795 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4252270835,v1:127.0.0.1:6803/4252270835] [v2:127.0.0.1:6804/4252270835,v1:127.0.0.1:6805/4252270835] exists,up 2c49bdea-4ffe-4e21-bfca-ce5ae9457936 2024-03-09T12:52:55.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:52:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:52:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:52:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:52:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:52:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T12:52:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:52:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T12:52:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:52:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/1 2024-03-09T12:52:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T12:52:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:52:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/1 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:52:55.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4d323b16-4a73-4575-b30b-5f6c502ec34e 2024-03-09T12:52:55.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4d323b16-4a73-4575-b30b-5f6c502ec34e' 2024-03-09T12:52:55.803 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 4d323b16-4a73-4575-b30b-5f6c502ec34e 2024-03-09T12:52:55.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:52:55.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnW+xllxzNMBAAxA1BFuAQwVXUlBAXODFOcg== 2024-03-09T12:52:55.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnW+xllxzNMBAAxA1BFuAQwVXUlBAXODFOcg=="}' 2024-03-09T12:52:55.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4d323b16-4a73-4575-b30b-5f6c502ec34e -i td/osd-recovery-scrub/1/new.json 2024-03-09T12:52:56.136 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:52:56.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/1/new.json 2024-03-09T12:52:56.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCnW+xllxzNMBAAxA1BFuAQwVXUlBAXODFOcg== --osd-uuid 4d323b16-4a73-4575-b30b-5f6c502ec34e 2024-03-09T12:52:56.179 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:56.181+0000 7f07ada13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:52:56.220 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:56.222+0000 7f07ada13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:52:56.221 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:56.223+0000 7f07ada13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:52:56.221 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:56.223+0000 7f07ada13640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-09T12:52:56.222 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:56.223+0000 7f07ada13640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-09T12:52:56.222 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:56.223+0000 7f07ada13640 -1 bluestore(td/osd-recovery-scrub/1/block) _read_bdev_label failed to open td/osd-recovery-scrub/1/block: (2) No such file or directory 2024-03-09T12:52:56.287 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:52:56.288+0000 7f07ada13640 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2024-03-09T12:53:02.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2024-03-09T12:53:02.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:53:02.111 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T12:53:02.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T12:53:02.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:53:02.462 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T12:53:02.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T12:53:02.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:53:02.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:53:02.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:53:02.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:53:02.492 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:02.493+0000 7f020c349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:02.494 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:02.495+0000 7f020c349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:02.495 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:02.496+0000 7f020c349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:02.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T12:53:02.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:53:02.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T12:53:02.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:53:02.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:53:02.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:02.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:53:02.734 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:53:02.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:02.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:53:03.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:03.025 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:03.026+0000 7f020c349640 -1 Falling back to public interface 2024-03-09T12:53:04.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:04.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:04.008 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:53:04.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:53:04.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:04.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:53:04.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:05.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:05.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:05.307 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:53:05.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:53:05.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:05.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:53:05.487 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:05.488+0000 7f020c349640 -1 osd.1 0 log_to_monitors true 2024-03-09T12:53:05.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:06.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:06.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:06.688 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:53:06.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:53:06.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:06.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:53:06.994 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2080227934,v1:127.0.0.1:6811/2080227934] [v2:127.0.0.1:6812/2080227934,v1:127.0.0.1:6813/2080227934] exists,up 4d323b16-4a73-4575-b30b-5f6c502ec34e 2024-03-09T12:53:06.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:53:06.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:53:06.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:53:06.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:53:06.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:53:06.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T12:53:06.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:53:06.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T12:53:06.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:53:06.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/2 2024-03-09T12:53:06.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T12:53:06.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:53:06.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:53:06.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:53:06.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2024-03-09T12:53:06.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2024-03-09T12:53:06.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:53:06.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:53:06.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T12:53:06.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:53:06.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:53:06.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:53:06.999 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:53:06.999 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:53:06.999 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:53:06.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/2 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:53:07.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0044f176-0a5b-4641-b45c-2990db249497 2024-03-09T12:53:07.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0044f176-0a5b-4641-b45c-2990db249497' 2024-03-09T12:53:07.001 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 0044f176-0a5b-4641-b45c-2990db249497 2024-03-09T12:53:07.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:53:07.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCzW+xlu0D+ABAAbL5VDxZaWQtGUYSdZbqt4A== 2024-03-09T12:53:07.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCzW+xlu0D+ABAAbL5VDxZaWQtGUYSdZbqt4A=="}' 2024-03-09T12:53:07.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0044f176-0a5b-4641-b45c-2990db249497 -i td/osd-recovery-scrub/2/new.json 2024-03-09T12:53:07.399 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:53:07.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/2/new.json 2024-03-09T12:53:07.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQCzW+xlu0D+ABAAbL5VDxZaWQtGUYSdZbqt4A== --osd-uuid 0044f176-0a5b-4641-b45c-2990db249497 2024-03-09T12:53:07.443 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:07.446+0000 7f1db439a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:07.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:07.486+0000 7f1db439a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:07.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:07.487+0000 7f1db439a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:07.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:07.488+0000 7f1db439a640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-09T12:53:07.485 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:07.488+0000 7f1db439a640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-09T12:53:07.486 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:07.488+0000 7f1db439a640 -1 bluestore(td/osd-recovery-scrub/2/block) _read_bdev_label failed to open td/osd-recovery-scrub/2/block: (2) No such file or directory 2024-03-09T12:53:07.542 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:07.545+0000 7f1db439a640 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2024-03-09T12:53:13.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2024-03-09T12:53:13.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:53:13.516 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T12:53:13.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T12:53:13.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:53:13.866 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T12:53:13.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T12:53:13.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:53:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:53:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:53:13.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:53:13.897 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:13.899+0000 7efedd499640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:13.898 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:13.901+0000 7efedd499640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:13.900 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:13.902+0000 7efedd499640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:14.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T12:53:14.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:53:14.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T12:53:14.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:53:14.143 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:53:14.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:53:14.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:14.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:53:14.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:14.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T12:53:14.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:14.424 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:14.427+0000 7efedd499640 -1 Falling back to public interface 2024-03-09T12:53:15.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:15.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:15.416 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:53:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:53:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T12:53:15.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:16.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:16.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:16.696 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:53:16.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:53:16.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:16.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T12:53:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:16.762+0000 7efedd499640 -1 osd.2 0 log_to_monitors true 2024-03-09T12:53:16.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:17.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:17.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:17.975 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:53:17.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:53:17.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:17.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T12:53:18.273 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1144195690,v1:127.0.0.1:6819/1144195690] [v2:127.0.0.1:6820/1144195690,v1:127.0.0.1:6821/1144195690] exists,up 0044f176-0a5b-4641-b45c-2990db249497 2024-03-09T12:53:18.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/3 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:53:18.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:53:18.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:53:18.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:53:18.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T12:53:18.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T12:53:18.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:53:18.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:53:18.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:53:18.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:53:18.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:53:18.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T12:53:18.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/3 2024-03-09T12:53:18.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:53:18.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=912f0a2c-051b-4c9c-975c-518f21afe063 2024-03-09T12:53:18.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 912f0a2c-051b-4c9c-975c-518f21afe063' 2024-03-09T12:53:18.281 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 912f0a2c-051b-4c9c-975c-518f21afe063 2024-03-09T12:53:18.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:53:18.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC+W+xlT2esERAA6CjSvxNiNMJ8JRT/oDhVpw== 2024-03-09T12:53:18.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC+W+xlT2esERAA6CjSvxNiNMJ8JRT/oDhVpw=="}' 2024-03-09T12:53:18.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 912f0a2c-051b-4c9c-975c-518f21afe063 -i td/osd-recovery-scrub/3/new.json 2024-03-09T12:53:18.639 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:53:18.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/3/new.json 2024-03-09T12:53:18.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQC+W+xlT2esERAA6CjSvxNiNMJ8JRT/oDhVpw== --osd-uuid 912f0a2c-051b-4c9c-975c-518f21afe063 2024-03-09T12:53:18.682 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:18.684+0000 7fe6b7e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:18.723 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:18.726+0000 7fe6b7e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:18.724 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:18.727+0000 7fe6b7e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:18.725 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:18.727+0000 7fe6b7e0c640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-09T12:53:18.725 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:18.727+0000 7fe6b7e0c640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-09T12:53:18.725 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:18.727+0000 7fe6b7e0c640 -1 bluestore(td/osd-recovery-scrub/3/block) _read_bdev_label failed to open td/osd-recovery-scrub/3/block: (2) No such file or directory 2024-03-09T12:53:18.798 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:18.801+0000 7fe6b7e0c640 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2024-03-09T12:53:24.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2024-03-09T12:53:24.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:53:24.648 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-03-09T12:53:24.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-09T12:53:24.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:53:25.001 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-03-09T12:53:25.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-09T12:53:25.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:53:25.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:53:25.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:53:25.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:53:25.034 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:25.034+0000 7f8ed99e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:25.036+0000 7f8ed99e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:25.037+0000 7f8ed99e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:25.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-09T12:53:25.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:53:25.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T12:53:25.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:53:25.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:53:25.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:25.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:53:25.303 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:53:25.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:25.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T12:53:25.556 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:25.559+0000 7f8ed99e1640 -1 Falling back to public interface 2024-03-09T12:53:25.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:26.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:26.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:26.575 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:53:26.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:53:26.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:26.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T12:53:26.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:27.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:27.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:27.857 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:53:27.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:53:27.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:27.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T12:53:27.894 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:27.893+0000 7f8ed99e1640 -1 osd.3 0 log_to_monitors true 2024-03-09T12:53:28.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:29.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:29.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:29.135 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:53:29.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:53:29.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:29.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T12:53:29.420 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1724226984,v1:127.0.0.1:6827/1724226984] [v2:127.0.0.1:6828/1724226984,v1:127.0.0.1:6829/1724226984] exists,up 912f0a2c-051b-4c9c-975c-518f21afe063 2024-03-09T12:53:29.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:53:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:53:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:53:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:53:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 4 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:53:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T12:53:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:53:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-09T12:53:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:53:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/4 2024-03-09T12:53:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T12:53:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:53:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:53:29.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:53:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/4' 2024-03-09T12:53:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/4/journal' 2024-03-09T12:53:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:53:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:53:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T12:53:29.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:53:29.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:53:29.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:53:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:53:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:53:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:53:29.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:53:29.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/4 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:53:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3d963016-e542-4896-99a6-9d6f4bf9c169 2024-03-09T12:53:29.428 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 3d963016-e542-4896-99a6-9d6f4bf9c169 2024-03-09T12:53:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 3d963016-e542-4896-99a6-9d6f4bf9c169' 2024-03-09T12:53:29.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:53:29.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJW+xlhoBxGhAAajU/ZGmjm5We95AlR3llCQ== 2024-03-09T12:53:29.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJW+xlhoBxGhAAajU/ZGmjm5We95AlR3llCQ=="}' 2024-03-09T12:53:29.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3d963016-e542-4896-99a6-9d6f4bf9c169 -i td/osd-recovery-scrub/4/new.json 2024-03-09T12:53:29.811 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T12:53:29.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/4/new.json 2024-03-09T12:53:29.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDJW+xlhoBxGhAAajU/ZGmjm5We95AlR3llCQ== --osd-uuid 3d963016-e542-4896-99a6-9d6f4bf9c169 2024-03-09T12:53:29.855 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:29.856+0000 7fde5901b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:29.896 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:29.897+0000 7fde5901b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:29.896 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:29.898+0000 7fde5901b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:29.897 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:29.898+0000 7fde5901b640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-09T12:53:29.897 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:29.898+0000 7fde5901b640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-09T12:53:29.897 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:29.898+0000 7fde5901b640 -1 bluestore(td/osd-recovery-scrub/4/block) _read_bdev_label failed to open td/osd-recovery-scrub/4/block: (2) No such file or directory 2024-03-09T12:53:29.970 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:29.972+0000 7fde5901b640 -1 bluestore(td/osd-recovery-scrub/4) _read_fsid unparsable uuid 2024-03-09T12:53:35.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/4/keyring 2024-03-09T12:53:35.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:53:35.964 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-03-09T12:53:35.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-09T12:53:35.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:53:36.319 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-03-09T12:53:36.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-09T12:53:36.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/4 --osd-journal=td/osd-recovery-scrub/4/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:53:36.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:53:36.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:53:36.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:53:36.349 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:36.351+0000 7f10f91bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:36.351 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:36.353+0000 7f10f91bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:36.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:36.354+0000 7f10f91bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:36.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-09T12:53:36.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:53:36.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-09T12:53:36.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:53:36.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:53:36.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:36.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:53:36.597 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:53:36.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:36.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T12:53:36.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:36.876 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:36.878+0000 7f10f91bb640 -1 Falling back to public interface 2024-03-09T12:53:37.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:37.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:37.870 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:53:37.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:53:37.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:37.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T12:53:38.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:39.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:39.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:39.155 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:53:39.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:53:39.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:39.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T12:53:39.200 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:39.202+0000 7f10f91bb640 -1 osd.4 0 log_to_monitors true 2024-03-09T12:53:39.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:40.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:40.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:40.470 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:53:40.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:53:40.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:40.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T12:53:40.824 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2990018560,v1:127.0.0.1:6835/2990018560] [v2:127.0.0.1:6836/2990018560,v1:127.0.0.1:6837/2990018560] exists,up 3d963016-e542-4896-99a6-9d6f4bf9c169 2024-03-09T12:53:40.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:53:40.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:53:40.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:53:40.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:53:40.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 5 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:53:40.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T12:53:40.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:53:40.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/5 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/5' 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/5/journal' 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:53:40.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:53:40.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:53:40.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:53:40.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T12:53:40.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/5 2024-03-09T12:53:40.831 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:53:40.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ab1063fc-971c-4b95-affa-842a332f8051 2024-03-09T12:53:40.831 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 ab1063fc-971c-4b95-affa-842a332f8051 2024-03-09T12:53:40.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 ab1063fc-971c-4b95-affa-842a332f8051' 2024-03-09T12:53:40.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:53:40.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUW+xlf+x/MhAAsgQZBpTMnJiAyLNd0b5/mw== 2024-03-09T12:53:40.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUW+xlf+x/MhAAsgQZBpTMnJiAyLNd0b5/mw=="}' 2024-03-09T12:53:40.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ab1063fc-971c-4b95-affa-842a332f8051 -i td/osd-recovery-scrub/5/new.json 2024-03-09T12:53:41.214 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T12:53:41.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/5/new.json 2024-03-09T12:53:41.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDUW+xlf+x/MhAAsgQZBpTMnJiAyLNd0b5/mw== --osd-uuid ab1063fc-971c-4b95-affa-842a332f8051 2024-03-09T12:53:41.258 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:41.259+0000 7f92f2eb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:41.299 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:41.300+0000 7f92f2eb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:41.300 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:41.301+0000 7f92f2eb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:41.300 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:41.302+0000 7f92f2eb5640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-09T12:53:41.300 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:41.302+0000 7f92f2eb5640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-09T12:53:41.300 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:41.302+0000 7f92f2eb5640 -1 bluestore(td/osd-recovery-scrub/5/block) _read_bdev_label failed to open td/osd-recovery-scrub/5/block: (2) No such file or directory 2024-03-09T12:53:41.365 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:41.367+0000 7f92f2eb5640 -1 bluestore(td/osd-recovery-scrub/5) _read_fsid unparsable uuid 2024-03-09T12:53:47.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/5/keyring 2024-03-09T12:53:47.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:53:47.300 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-03-09T12:53:47.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-09T12:53:47.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:53:47.656 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-03-09T12:53:47.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-09T12:53:47.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/5 --osd-journal=td/osd-recovery-scrub/5/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:53:47.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:53:47.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:53:47.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:53:47.686 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:47.688+0000 7f1d9280f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:47.784 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:47.690+0000 7f1d9280f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:47.784 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:47.691+0000 7f1d9280f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:47.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-09T12:53:47.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:53:47.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T12:53:47.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:53:47.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:53:47.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:47.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:53:47.932 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:53:47.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:47.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T12:53:48.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:48.210 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:48.212+0000 7f1d9280f640 -1 Falling back to public interface 2024-03-09T12:53:49.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:49.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:49.205 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:53:49.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:53:49.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:49.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T12:53:49.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:50.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:50.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:50.491 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:53:50.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:53:50.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:50.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T12:53:50.577 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:50.578+0000 7f1d9280f640 -1 osd.5 0 log_to_monitors true 2024-03-09T12:53:50.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:53:51.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:53:51.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:51.787 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:53:51.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:53:51.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:51.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T12:53:52.094 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1659050127,v1:127.0.0.1:6843/1659050127] [v2:127.0.0.1:6844/1659050127,v1:127.0.0.1:6845/1659050127] exists,up ab1063fc-971c-4b95-affa-842a332f8051 2024-03-09T12:53:52.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:53:52.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 6 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/6 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/6' 2024-03-09T12:53:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/6/journal' 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:53:52.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:53:52.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T12:53:52.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T12:53:52.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:53:52.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:53:52.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:53:52.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:53:52.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:53:52.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T12:53:52.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/6 2024-03-09T12:53:52.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:53:52.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d3d6262a-df2a-48cd-8101-875ee764c4da 2024-03-09T12:53:52.102 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 d3d6262a-df2a-48cd-8101-875ee764c4da 2024-03-09T12:53:52.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 d3d6262a-df2a-48cd-8101-875ee764c4da' 2024-03-09T12:53:52.102 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:53:52.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDgW+xl5/YHBxAAyZBo+Fzacp3jdGdQKhYhng== 2024-03-09T12:53:52.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDgW+xl5/YHBxAAyZBo+Fzacp3jdGdQKhYhng=="}' 2024-03-09T12:53:52.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d3d6262a-df2a-48cd-8101-875ee764c4da -i td/osd-recovery-scrub/6/new.json 2024-03-09T12:53:52.500 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T12:53:52.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/6/new.json 2024-03-09T12:53:52.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDgW+xl5/YHBxAAyZBo+Fzacp3jdGdQKhYhng== --osd-uuid d3d6262a-df2a-48cd-8101-875ee764c4da 2024-03-09T12:53:52.544 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:52.545+0000 7f77b43bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:52.559 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:52.561+0000 7f77b43bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:52.560 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:52.562+0000 7f77b43bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:52.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:52.563+0000 7f77b43bd640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-09T12:53:52.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:52.563+0000 7f77b43bd640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-09T12:53:52.561 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:52.563+0000 7f77b43bd640 -1 bluestore(td/osd-recovery-scrub/6/block) _read_bdev_label failed to open td/osd-recovery-scrub/6/block: (2) No such file or directory 2024-03-09T12:53:52.668 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:52.670+0000 7f77b43bd640 -1 bluestore(td/osd-recovery-scrub/6) _read_fsid unparsable uuid 2024-03-09T12:53:58.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/6/keyring 2024-03-09T12:53:58.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:53:58.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-09T12:53:58.672 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-03-09T12:53:58.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:53:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-09T12:53:59.025 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-03-09T12:53:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/6 --osd-journal=td/osd-recovery-scrub/6/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:53:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:53:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:53:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:53:59.055 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:59.057+0000 7f155e80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:59.057 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:59.059+0000 7f155e80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:59.058 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:59.061+0000 7f155e80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:53:59.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-09T12:53:59.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:53:59.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-09T12:53:59.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:53:59.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:53:59.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:53:59.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:53:59.302 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:53:59.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:53:59.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T12:53:59.584 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:53:59.587+0000 7f155e80c640 -1 Falling back to public interface 2024-03-09T12:53:59.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:54:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:54:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:54:00.597 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:54:00.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:54:00.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:54:00.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T12:54:00.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:54:01.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:54:01.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:54:01.881 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:54:01.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:54:01.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:54:01.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T12:54:01.924 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:54:01.926+0000 7f155e80c640 -1 osd.6 0 log_to_monitors true 2024-03-09T12:54:02.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:54:03.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:54:03.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:54:03.162 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:54:03.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:54:03.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:54:03.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T12:54:03.449 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/107471147,v1:127.0.0.1:6851/107471147] [v2:127.0.0.1:6852/107471147,v1:127.0.0.1:6853/107471147] exists,up d3d6262a-df2a-48cd-8101-875ee764c4da 2024-03-09T12:54:03.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:54:03.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:54:03.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:249: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:251: TEST_recovery_scrub_2: run_osd td/osd-recovery-scrub 7 --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-scrub 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-scrub/7 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/7' 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/7/journal' 2024-03-09T12:54:03.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:54:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2024-03-09T12:54:03.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2024-03-09T12:54:03.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:54:03.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:54:03.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:54:03.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:54:03.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:54:03.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2024-03-09T12:54:03.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-scrub/7 2024-03-09T12:54:03.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:54:03.456 INFO:tasks.workunit.client.0.smithi162.stdout:add osd7 a476811f-8621-44cf-9ef7-4e68e821236e 2024-03-09T12:54:03.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a476811f-8621-44cf-9ef7-4e68e821236e 2024-03-09T12:54:03.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 a476811f-8621-44cf-9ef7-4e68e821236e' 2024-03-09T12:54:03.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:54:03.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDrW+xl2IEoHBAAQmt+h3ZivPPODOADk6gexg== 2024-03-09T12:54:03.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDrW+xl2IEoHBAAQmt+h3ZivPPODOADk6gexg=="}' 2024-03-09T12:54:03.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a476811f-8621-44cf-9ef7-4e68e821236e -i td/osd-recovery-scrub/7/new.json 2024-03-09T12:54:03.791 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T12:54:03.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-scrub/7/new.json 2024-03-09T12:54:03.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQDrW+xl2IEoHBAAQmt+h3ZivPPODOADk6gexg== --osd-uuid a476811f-8621-44cf-9ef7-4e68e821236e 2024-03-09T12:54:03.836 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:54:03.838+0000 7f8aa5048640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:54:03.842 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:54:03.844+0000 7f8aa5048640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:54:03.843 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:54:03.845+0000 7f8aa5048640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:54:03.843 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:54:03.846+0000 7f8aa5048640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-09T12:54:03.843 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:54:03.846+0000 7f8aa5048640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-09T12:54:03.843 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:54:03.846+0000 7f8aa5048640 -1 bluestore(td/osd-recovery-scrub/7/block) _read_bdev_label failed to open td/osd-recovery-scrub/7/block: (2) No such file or directory 2024-03-09T12:54:03.933 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:54:03.936+0000 7f8aa5048640 -1 bluestore(td/osd-recovery-scrub/7) _read_fsid unparsable uuid 2024-03-09T12:54:10.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-scrub/7/keyring 2024-03-09T12:54:10.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:54:10.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-09T12:54:10.264 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd7 key to auth repository 2024-03-09T12:54:10.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-scrub/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:54:10.621 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.7 2024-03-09T12:54:10.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-09T12:54:10.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=c22230a3-60b8-4dc8-8ae7-2366bc50ab0a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/7 --osd-journal=td/osd-recovery-scrub/7/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.44981/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=true --osd_recovery_sleep=10 --osd-scrub-interval-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2024-03-09T12:54:10.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:54:10.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:54:10.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:54:10.651 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:54:10.653+0000 7f2e2481c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:54:10.652 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:54:10.655+0000 7f2e2481c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:54:10.653 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:54:10.656+0000 7f2e2481c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:54:10.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-09T12:54:10.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:54:10.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-09T12:54:10.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:54:10.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:54:10.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:54:10.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:54:10.899 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:54:10.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:54:10.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T12:54:11.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:54:11.185 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:54:11.188+0000 7f2e2481c640 -1 Falling back to public interface 2024-03-09T12:54:12.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:54:12.177 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:54:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:54:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:54:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:54:12.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T12:54:12.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:54:13.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:54:13.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:54:13.460 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:54:13.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:54:13.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:54:13.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T12:54:13.653 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:54:13.656+0000 7f2e2481c640 -1 osd.7 0 log_to_monitors true 2024-03-09T12:54:13.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:54:14.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:54:14.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:54:14.754 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:54:14.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:54:14.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:54:14.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T12:54:15.031 INFO:tasks.workunit.client.0.smithi162.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2488100664,v1:127.0.0.1:6859/2488100664] [v2:127.0.0.1:6860/2488100664,v1:127.0.0.1:6861/2488100664] exists,up a476811f-8621-44cf-9ef7-4e68e821236e 2024-03-09T12:54:15.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:54:15.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:54:15.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:54:15.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:256: TEST_recovery_scrub_2: create_pool test 32 32 2024-03-09T12:54:15.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 32 32 2024-03-09T12:54:15.493 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'test' created 2024-03-09T12:54:15.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T12:54:16.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:257: TEST_recovery_scrub_2: wait_for_clean 2024-03-09T12:54:16.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T12:54:16.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T12:54:16.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T12:54:16.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T12:54:16.507 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T12:54:16.508 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T12:54:16.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T12:54:16.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T12:54:16.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T12:54:16.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T12:54:16.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T12:54:16.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T12:54:16.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T12:54:16.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T12:54:16.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T12:54:17.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T12:54:17.025 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T12:54:17.025 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T12:54:17.025 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T12:54:17.025 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-03-09T12:54:17.025 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T12:54:17.025 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T12:54:17.025 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-03-09T12:54:17.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T12:54:17.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:54:17.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T12:54:17.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-03-09T12:54:17.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-03-09T12:54:17.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-03-09T12:54:17.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:54:17.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T12:54:17.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-09T12:54:17.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-09T12:54:17.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2024-03-09T12:54:17.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:54:17.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T12:54:17.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-09T12:54:17.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-09T12:54:17.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453' 2024-03-09T12:54:17.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:54:17.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T12:54:17.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345931 2024-03-09T12:54:17.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345931 2024-03-09T12:54:17.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931' 2024-03-09T12:54:17.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:54:17.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T12:54:17.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182409 2024-03-09T12:54:17.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182409 2024-03-09T12:54:17.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409' 2024-03-09T12:54:17.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:54:17.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T12:54:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-03-09T12:54:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-03-09T12:54:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409 5-128849018887' 2024-03-09T12:54:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:54:18.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T12:54:18.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855364 2024-03-09T12:54:18.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855364 2024-03-09T12:54:18.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855364' 2024-03-09T12:54:18.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:54:18.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T12:54:18.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-09T12:54:18.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-09T12:54:18.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509453 3-85899345931 4-107374182409 5-128849018887 6-150323855364 7-171798691842' 2024-03-09T12:54:18.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:54:18.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-03-09T12:54:18.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:54:18.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T12:54:18.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-03-09T12:54:18.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:54:18.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-03-09T12:54:18.545 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836498 2024-03-09T12:54:18.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-03-09T12:54:18.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:54:18.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-03-09T12:54:18.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:54:18.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-09T12:54:18.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:54:18.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T12:54:18.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-09T12:54:18.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:54:18.911 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672976 2024-03-09T12:54:18.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-09T12:54:18.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-09T12:54:18.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T12:54:19.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672976 2024-03-09T12:54:19.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:54:20.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T12:54:20.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T12:54:20.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672976 2024-03-09T12:54:20.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:54:20.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-09T12:54:20.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:54:20.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T12:54:20.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-09T12:54:20.515 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:54:20.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-09T12:54:20.516 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509453 2024-03-09T12:54:20.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-09T12:54:20.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T12:54:20.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2024-03-09T12:54:20.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:54:20.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345931 2024-03-09T12:54:20.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:54:20.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T12:54:20.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345931 2024-03-09T12:54:20.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:54:20.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345931 2024-03-09T12:54:20.792 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345931 2024-03-09T12:54:20.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-03-09T12:54:20.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T12:54:21.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345931 -lt 85899345931 2024-03-09T12:54:21.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:54:21.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182409 2024-03-09T12:54:21.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:54:21.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T12:54:21.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182409 2024-03-09T12:54:21.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:54:21.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182409 2024-03-09T12:54:21.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2024-03-09T12:54:21.070 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182409 2024-03-09T12:54:21.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T12:54:21.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182410 -lt 107374182409 2024-03-09T12:54:21.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:54:21.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018887 2024-03-09T12:54:21.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:54:21.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T12:54:21.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018887 2024-03-09T12:54:21.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:54:21.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-03-09T12:54:21.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-03-09T12:54:21.360 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 128849018887 2024-03-09T12:54:21.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T12:54:21.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-03-09T12:54:21.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:54:21.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855364 2024-03-09T12:54:21.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:54:21.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T12:54:21.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855364 2024-03-09T12:54:21.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:54:21.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855364 2024-03-09T12:54:21.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855364' 2024-03-09T12:54:21.637 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 150323855364 2024-03-09T12:54:21.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T12:54:21.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855364 2024-03-09T12:54:21.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:54:21.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691842 2024-03-09T12:54:21.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:54:21.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T12:54:21.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691842 2024-03-09T12:54:21.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:54:21.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-09T12:54:21.914 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691842 2024-03-09T12:54:21.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691842' 2024-03-09T12:54:21.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T12:54:22.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691842 2024-03-09T12:54:22.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T12:54:22.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:54:22.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:54:22.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-03-09T12:54:22.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T12:54:22.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T12:54:22.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T12:54:22.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T12:54:22.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T12:54:22.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T12:54:22.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T12:54:22.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=25 2024-03-09T12:54:22.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T12:54:22.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:54:22.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:54:23.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 25 = 32 2024-03-09T12:54:23.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 25 '!=' -1 2024-03-09T12:54:23.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T12:54:23.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=25 2024-03-09T12:54:23.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T12:54:23.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T12:54:23.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T12:54:23.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T12:54:23.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T12:54:23.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T12:54:23.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T12:54:23.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T12:54:23.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T12:54:23.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T12:54:23.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-03-09T12:54:23.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T12:54:23.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:54:23.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:54:23.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-03-09T12:54:23.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T12:54:23.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T12:54:23.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: ceph osd dump 2024-03-09T12:54:23.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: grep '^pool.*['\'']test['\'']' 2024-03-09T12:54:23.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: awk '{ print $2 }' 2024-03-09T12:54:24.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:258: TEST_recovery_scrub_2: poolid=1 2024-03-09T12:54:24.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:260: TEST_recovery_scrub_2: dd if=/dev/urandom of=testdata.44981 bs=1M count=50 2024-03-09T12:54:24.271 INFO:tasks.workunit.client.0.smithi162.stderr:50+0 records in 2024-03-09T12:54:24.271 INFO:tasks.workunit.client.0.smithi162.stderr:50+0 records out 2024-03-09T12:54:24.271 INFO:tasks.workunit.client.0.smithi162.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.142186 s, 369 MB/s 2024-03-09T12:54:24.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: seq 1 40 2024-03-09T12:54:24.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:24.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj1 testdata.44981 2024-03-09T12:54:25.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:25.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj2 testdata.44981 2024-03-09T12:54:27.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:27.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj3 testdata.44981 2024-03-09T12:54:28.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:28.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj4 testdata.44981 2024-03-09T12:54:30.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:30.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj5 testdata.44981 2024-03-09T12:54:31.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:31.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj6 testdata.44981 2024-03-09T12:54:33.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:33.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj7 testdata.44981 2024-03-09T12:54:34.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:34.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj8 testdata.44981 2024-03-09T12:54:36.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:36.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj9 testdata.44981 2024-03-09T12:54:37.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:37.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj10 testdata.44981 2024-03-09T12:54:39.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:39.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj11 testdata.44981 2024-03-09T12:54:40.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:40.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj12 testdata.44981 2024-03-09T12:54:42.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:42.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj13 testdata.44981 2024-03-09T12:54:43.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:43.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj14 testdata.44981 2024-03-09T12:54:45.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:45.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj15 testdata.44981 2024-03-09T12:54:46.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:46.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj16 testdata.44981 2024-03-09T12:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj17 testdata.44981 2024-03-09T12:54:49.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:49.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj18 testdata.44981 2024-03-09T12:54:51.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:51.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj19 testdata.44981 2024-03-09T12:54:52.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:52.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj20 testdata.44981 2024-03-09T12:54:54.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:54.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj21 testdata.44981 2024-03-09T12:54:55.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:55.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj22 testdata.44981 2024-03-09T12:54:57.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:57.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj23 testdata.44981 2024-03-09T12:54:59.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:54:59.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj24 testdata.44981 2024-03-09T12:55:00.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:00.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj25 testdata.44981 2024-03-09T12:55:02.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:02.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj26 testdata.44981 2024-03-09T12:55:03.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:03.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj27 testdata.44981 2024-03-09T12:55:05.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:05.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj28 testdata.44981 2024-03-09T12:55:06.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:06.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj29 testdata.44981 2024-03-09T12:55:08.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:08.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj30 testdata.44981 2024-03-09T12:55:09.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:09.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj31 testdata.44981 2024-03-09T12:55:11.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:11.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj32 testdata.44981 2024-03-09T12:55:12.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:12.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj33 testdata.44981 2024-03-09T12:55:14.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:14.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj34 testdata.44981 2024-03-09T12:55:15.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:15.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj35 testdata.44981 2024-03-09T12:55:17.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:17.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj36 testdata.44981 2024-03-09T12:55:18.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:18.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj37 testdata.44981 2024-03-09T12:55:20.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:20.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj38 testdata.44981 2024-03-09T12:55:21.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:21.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj39 testdata.44981 2024-03-09T12:55:23.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:261: TEST_recovery_scrub_2: for i in $(seq 1 $OBJECTS) 2024-03-09T12:55:23.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:263: TEST_recovery_scrub_2: rados -p test put obj40 testdata.44981 2024-03-09T12:55:24.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:265: TEST_recovery_scrub_2: rm -f testdata.44981 2024-03-09T12:55:24.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:267: TEST_recovery_scrub_2: ceph osd pool set test size 3 2024-03-09T12:55:25.055 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 1 size to 3 2024-03-09T12:55:25.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:269: TEST_recovery_scrub_2: ceph pg dump pgs 2024-03-09T12:55:25.333 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T12:55:25.333 INFO:tasks.workunit.client.0.smithi162.stdout:1.1f 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-03-09T12:54:17.016426+0000 50'26 50:59 [5] 5 [5] 5 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.333 INFO:tasks.workunit.client.0.smithi162.stdout:1.1e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:17.016572+0000 50'13 50:47 [3] 3 [3] 3 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.334 INFO:tasks.workunit.client.0.smithi162.stdout:1.1d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:18.023388+0000 50'13 50:31 [1] 1 [1] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.334 INFO:tasks.workunit.client.0.smithi162.stdout:1.b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:17.926752+0000 50'13 50:34 [1] 1 [1] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.334 INFO:tasks.workunit.client.0.smithi162.stdout:1.a 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:17.926819+0000 50'13 50:34 [1] 1 [1] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.334 INFO:tasks.workunit.client.0.smithi162.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:54:17.926818+0000 0'0 50:21 [1] 1 [1] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.334 INFO:tasks.workunit.client.0.smithi162.stdout:1.8 2 0 0 0 0 104857600 0 0 26 0 26 active+clean 2024-03-09T12:54:16.881262+0000 50'26 50:50 [5] 5 [5] 5 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.334 INFO:tasks.workunit.client.0.smithi162.stdout:1.7 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:17.528155+0000 50'13 50:37 [6] 6 [6] 6 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.334 INFO:tasks.workunit.client.0.smithi162.stdout:1.6 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:17.436024+0000 50'13 50:37 [6] 6 [6] 6 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.334 INFO:tasks.workunit.client.0.smithi162.stdout:1.5 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-03-09T12:54:17.506675+0000 50'39 50:64 [4] 4 [4] 4 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.334 INFO:tasks.workunit.client.0.smithi162.stdout:1.4 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-03-09T12:54:16.881233+0000 50'52 50:79 [3] 3 [3] 3 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.334 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 4 0 0 0 0 209715200 0 0 52 0 52 active+clean 2024-03-09T12:54:17.376980+0000 50'52 50:77 [4] 4 [4] 4 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.334 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:16.423582+0000 50'13 50:41 [0] 0 [0] 0 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.334 INFO:tasks.workunit.client.0.smithi162.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:54:17.016056+0000 0'0 50:25 [3] 3 [3] 3 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.335 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:16.881270+0000 50'13 50:46 [5] 5 [5] 5 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.335 INFO:tasks.workunit.client.0.smithi162.stdout:1.1c 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:16.503916+0000 50'13 50:33 [0] 0 [0] 0 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.335 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:16.881298+0000 50'13 50:44 [7] 7 [7] 7 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.335 INFO:tasks.workunit.client.0.smithi162.stdout:1.d 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:17.016075+0000 50'13 50:34 [7] 7 [7] 7 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.335 INFO:tasks.workunit.client.0.smithi162.stdout:1.e 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:18.023066+0000 50'13 50:33 [1] 1 [1] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.335 INFO:tasks.workunit.client.0.smithi162.stdout:1.f 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:17.436024+0000 50'13 50:34 [6] 6 [6] 6 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.335 INFO:tasks.workunit.client.0.smithi162.stdout:1.10 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:17.016071+0000 50'13 50:36 [5] 5 [5] 5 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.335 INFO:tasks.workunit.client.0.smithi162.stdout:1.11 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:16.503827+0000 50'13 50:35 [0] 0 [0] 0 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.335 INFO:tasks.workunit.client.0.smithi162.stdout:1.12 2 0 0 0 0 65011712 0 0 16 0 16 active+clean 2024-03-09T12:54:16.423930+0000 50'16 50:37 [0] 0 [0] 0 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.335 INFO:tasks.workunit.client.0.smithi162.stdout:1.13 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:17.506675+0000 50'13 50:34 [4] 4 [4] 4 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.335 INFO:tasks.workunit.client.0.smithi162.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:54:16.856784+0000 0'0 50:21 [2] 2 [2] 2 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.335 INFO:tasks.workunit.client.0.smithi162.stdout:1.15 3 0 0 0 0 157286400 0 0 39 0 39 active+clean 2024-03-09T12:54:18.023331+0000 50'39 50:58 [1] 1 [1] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.335 INFO:tasks.workunit.client.0.smithi162.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:54:17.528503+0000 0'0 50:20 [6] 6 [6] 6 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.336 INFO:tasks.workunit.client.0.smithi162.stdout:1.17 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:16.503796+0000 50'13 50:34 [0] 0 [0] 0 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.336 INFO:tasks.workunit.client.0.smithi162.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:54:16.881662+0000 0'0 50:22 [5] 5 [5] 5 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.336 INFO:tasks.workunit.client.0.smithi162.stdout:1.19 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:17.016428+0000 50'13 50:36 [3] 3 [3] 3 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.336 INFO:tasks.workunit.client.0.smithi162.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:54:18.023507+0000 0'0 50:19 [1] 1 [1] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.336 INFO:tasks.workunit.client.0.smithi162.stdout:1.1b 1 0 0 0 0 52428800 0 0 13 0 13 active+clean 2024-03-09T12:54:16.503964+0000 50'13 50:34 [0] 0 [0] 0 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:25.336 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:55:25.336 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T12:55:25.336 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:55:25.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:275: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active 2024-03-09T12:55:25.448 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T12:55:25.448 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_recovery_max_active": "0" 2024-03-09T12:55:25.448 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:55:25.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:277: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_hdd 2024-03-09T12:55:25.692 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T12:55:25.692 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_recovery_max_active_hdd": "3" 2024-03-09T12:55:25.692 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:55:25.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:278: TEST_recovery_scrub_2: ceph tell osd.1 config get osd_recovery_max_active_ssd 2024-03-09T12:55:25.986 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T12:55:25.987 INFO:tasks.workunit.client.0.smithi162.stdout: "osd_recovery_max_active_ssd": "10" 2024-03-09T12:55:25.987 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:55:25.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:281: TEST_recovery_scrub_2: count=0 2024-03-09T12:55:25.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-09T12:55:25.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-09T12:55:25.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-09T12:55:25.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-09T12:55:25.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-09T12:55:26.413 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:55:26.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-03-09T12:55:26.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-03-09T12:55:28.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 0 -eq 10 2024-03-09T12:55:28.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 0 + 1 2024-03-09T12:55:28.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=1 2024-03-09T12:55:28.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-09T12:55:28.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-09T12:55:28.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-09T12:55:28.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-09T12:55:28.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-09T12:55:28.698 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:55:28.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 0 -ge 2 2024-03-09T12:55:28.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-03-09T12:55:30.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 1 -eq 10 2024-03-09T12:55:30.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 1 + 1 2024-03-09T12:55:30.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=2 2024-03-09T12:55:30.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-09T12:55:30.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-09T12:55:30.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-09T12:55:30.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-09T12:55:30.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-09T12:55:30.987 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:55:30.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 1 -ge 2 2024-03-09T12:55:30.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:290: TEST_recovery_scrub_2: sleep 2 2024-03-09T12:55:33.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:291: TEST_recovery_scrub_2: test 2 -eq 10 2024-03-09T12:55:33.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: expr 2 + 1 2024-03-09T12:55:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:296: TEST_recovery_scrub_2: count=3 2024-03-09T12:55:33.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:282: TEST_recovery_scrub_2: true 2024-03-09T12:55:33.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: ceph --format json pg dump pgs 2024-03-09T12:55:33.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: jq '.pg_stats | [.[].state]' 2024-03-09T12:55:33.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: grep recovering 2024-03-09T12:55:33.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:287: TEST_recovery_scrub_2: wc -l 2024-03-09T12:55:33.274 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:55:33.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:286: TEST_recovery_scrub_2: test 2 -ge 2 2024-03-09T12:55:33.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:288: TEST_recovery_scrub_2: break 2024-03-09T12:55:33.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:298: TEST_recovery_scrub_2: ceph pg dump pgs 2024-03-09T12:55:33.544 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T12:55:33.545 INFO:tasks.workunit.client.0.smithi162.stdout:1.1f 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-09T12:55:27.090992+0000 50'26 53:70 [5,0,2] 5 [5,0,2] 5 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.545 INFO:tasks.workunit.client.0.smithi162.stdout:1.1e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:27.090997+0000 50'13 53:57 [3,0,7] 3 [3,0,7] 3 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.545 INFO:tasks.workunit.client.0.smithi162.stdout:1.1d 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-09T12:55:27.225193+0000 50'26 53:54 [1,3,0] 1 [1,3,0] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.545 INFO:tasks.workunit.client.0.smithi162.stdout:1.b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:26.933584+0000 50'13 53:44 [1,0,2] 1 [1,0,2] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.545 INFO:tasks.workunit.client.0.smithi162.stdout:1.a 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:27.351585+0000 50'13 53:44 [1,6,2] 1 [1,6,2] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.545 INFO:tasks.workunit.client.0.smithi162.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:55:26.896078+0000 0'0 53:33 [1,4,5] 1 [1,4,5] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:33.545 INFO:tasks.workunit.client.0.smithi162.stdout:1.8 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-09T12:55:27.058159+0000 50'26 53:60 [5,4,3] 5 [5,4,3] 5 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.545 INFO:tasks.workunit.client.0.smithi162.stdout:1.7 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-03-09T12:55:26.850639+0000 50'13 53:49 [6,7,3] 6 [6,7,3] 6 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.545 INFO:tasks.workunit.client.0.smithi162.stdout:1.6 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:26.998393+0000 50'13 53:48 [6,7,3] 6 [6,7,3] 6 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.545 INFO:tasks.workunit.client.0.smithi162.stdout:1.5 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-03-09T12:55:27.536397+0000 50'39 53:74 [4,2,6] 4 [4,2,6] 4 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.545 INFO:tasks.workunit.client.0.smithi162.stdout:1.4 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-03-09T12:55:27.351659+0000 50'52 53:89 [3,0,6] 3 [3,0,6] 3 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.545 INFO:tasks.workunit.client.0.smithi162.stdout:1.3 4 0 8 0 0 209715200 0 0 52 0 52 active+recovery_wait+degraded 2024-03-09T12:55:27.057393+0000 50'52 53:87 [4,1,2] 4 [4,1,2] 4 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:1.2 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:27.187269+0000 50'13 53:51 [0,7,1] 0 [0,7,1] 0 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:55:27.057260+0000 0'0 53:36 [3,0,4] 3 [3,0,4] 3 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:1.1 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:27.272332+0000 50'13 53:57 [5,0,4] 5 [5,0,4] 5 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:1.1c 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:27.225341+0000 50'13 53:43 [0,3,4] 0 [0,3,4] 0 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:27.187090+0000 50'13 53:54 [7,0,2] 7 [7,0,2] 7 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:1.d 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:27.058758+0000 50'13 53:43 [7,4,2] 7 [7,4,2] 7 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:1.e 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:26.895766+0000 50'13 53:42 [1,4,2] 1 [1,4,2] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:1.f 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:27.186879+0000 50'13 53:44 [6,0,3] 6 [6,0,3] 6 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:1.10 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:27.351744+0000 50'13 53:45 [5,7,6] 5 [5,7,6] 5 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:1.11 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:27.057714+0000 50'13 53:44 [0,1,3] 0 [0,1,3] 0 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:1.12 2 0 4 0 0 104857600 0 0 26 0 26 active+recovery_wait+degraded 2024-03-09T12:55:27.351516+0000 50'26 53:57 [0,1,6] 0 [0,1,6] 0 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:1.13 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:27.187193+0000 50'13 53:44 [4,6,3] 4 [4,6,3] 4 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.546 INFO:tasks.workunit.client.0.smithi162.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:55:26.933395+0000 0'0 53:32 [2,0,4] 2 [2,0,4] 2 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:33.547 INFO:tasks.workunit.client.0.smithi162.stdout:1.15 3 0 6 0 0 157286400 0 0 39 0 39 active+recovery_wait+degraded 2024-03-09T12:55:27.057286+0000 50'39 53:68 [1,5,3] 1 [1,5,3] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.547 INFO:tasks.workunit.client.0.smithi162.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:55:26.178615+0000 0'0 53:32 [6,0,7] 6 [6,0,7] 6 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:33.547 INFO:tasks.workunit.client.0.smithi162.stdout:1.17 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:27.401927+0000 50'13 53:43 [0,4,5] 0 [0,4,5] 0 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.547 INFO:tasks.workunit.client.0.smithi162.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:55:27.057286+0000 0'0 53:33 [5,2,3] 5 [5,2,3] 5 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:33.547 INFO:tasks.workunit.client.0.smithi162.stdout:1.19 1 0 2 0 0 52428800 0 0 13 0 13 active+recovery_wait+degraded 2024-03-09T12:55:27.272056+0000 50'13 53:45 [3,0,4] 3 [3,0,4] 3 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.547 INFO:tasks.workunit.client.0.smithi162.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-03-09T12:55:26.741017+0000 0'0 53:30 [1,7,4] 1 [1,7,4] 1 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 periodic scrub scheduled @ 2024-03-10T12:54:15.378828+0000 0 0 2024-03-09T12:55:33.547 INFO:tasks.workunit.client.0.smithi162.stdout:1.1b 1 0 2 0 0 52428800 0 0 13 0 13 active+recovering+degraded 2024-03-09T12:55:26.851803+0000 50'13 53:44 [0,1,2] 0 [0,1,2] 0 0'0 2024-03-09T12:54:15.378828+0000 0'0 2024-03-09T12:54:15.378828+0000 0 0 no scrub is scheduled 0 0 2024-03-09T12:55:33.547 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:55:33.547 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T12:55:33.547 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:55:33.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:300: TEST_recovery_scrub_2: pids= 2024-03-09T12:55:33.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:301: TEST_recovery_scrub_2: recov_scrub_count=0 2024-03-09T12:55:33.556 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: expr 32 - 1 2024-03-09T12:55:33.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: seq 0 31 2024-03-09T12:55:33.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 0 2024-03-09T12:55:33.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.0 2024-03-09T12:55:33.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66531"' 2024-03-09T12:55:33.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66531' 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.0 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 1 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66534: /' 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66536"' 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66536' 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 2 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66541: /' 2024-03-09T12:55:33.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.2 2024-03-09T12:55:33.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66544"' 2024-03-09T12:55:33.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66544' 2024-03-09T12:55:33.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66548: /' 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 3 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.2 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.3 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66550"' 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66550' 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.3 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66553: /' 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 4 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.4 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66559"' 2024-03-09T12:55:33.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66559' 2024-03-09T12:55:33.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.4 2024-03-09T12:55:33.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 5 2024-03-09T12:55:33.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.5 2024-03-09T12:55:33.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66561: /' 2024-03-09T12:55:33.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66564"' 2024-03-09T12:55:33.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66564' 2024-03-09T12:55:33.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.5 2024-03-09T12:55:33.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66569: /' 2024-03-09T12:55:33.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 6 2024-03-09T12:55:33.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.6 2024-03-09T12:55:33.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66573"' 2024-03-09T12:55:33.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66573' 2024-03-09T12:55:33.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 7 2024-03-09T12:55:33.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.7 2024-03-09T12:55:33.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66576"' 2024-03-09T12:55:33.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66576' 2024-03-09T12:55:33.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66577: /' 2024-03-09T12:55:33.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.7 2024-03-09T12:55:33.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66580: /' 2024-03-09T12:55:33.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 8 2024-03-09T12:55:33.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.8 2024-03-09T12:55:33.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66584"' 2024-03-09T12:55:33.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66584' 2024-03-09T12:55:33.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 9 2024-03-09T12:55:33.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.8 2024-03-09T12:55:33.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.9 2024-03-09T12:55:33.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66589"' 2024-03-09T12:55:33.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66589' 2024-03-09T12:55:33.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.9 2024-03-09T12:55:33.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66594: /' 2024-03-09T12:55:33.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.6 2024-03-09T12:55:33.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 10 2024-03-09T12:55:33.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.a 2024-03-09T12:55:33.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66599"' 2024-03-09T12:55:33.588 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66599' 2024-03-09T12:55:33.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66587: /' 2024-03-09T12:55:33.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 11 2024-03-09T12:55:33.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.b 2024-03-09T12:55:33.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66603"' 2024-03-09T12:55:33.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66603' 2024-03-09T12:55:33.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 12 2024-03-09T12:55:33.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.c 2024-03-09T12:55:33.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66607"' 2024-03-09T12:55:33.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66607' 2024-03-09T12:55:33.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66606: /' 2024-03-09T12:55:33.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 13 2024-03-09T12:55:33.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.d 2024-03-09T12:55:33.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66611"' 2024-03-09T12:55:33.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66611' 2024-03-09T12:55:33.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66610: /' 2024-03-09T12:55:33.625 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 14 2024-03-09T12:55:33.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.e 2024-03-09T12:55:33.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.a 2024-03-09T12:55:33.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66615"' 2024-03-09T12:55:33.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66615' 2024-03-09T12:55:33.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.b 2024-03-09T12:55:33.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.c 2024-03-09T12:55:33.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 15 2024-03-09T12:55:33.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.f 2024-03-09T12:55:33.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66624"' 2024-03-09T12:55:33.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66624' 2024-03-09T12:55:33.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66614: /' 2024-03-09T12:55:33.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 16 2024-03-09T12:55:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.10 2024-03-09T12:55:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66630"' 2024-03-09T12:55:33.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66630' 2024-03-09T12:55:33.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.e 2024-03-09T12:55:33.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66621: /' 2024-03-09T12:55:33.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66623: /' 2024-03-09T12:55:33.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.d 2024-03-09T12:55:33.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 17 2024-03-09T12:55:33.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.11 2024-03-09T12:55:33.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66638"' 2024-03-09T12:55:33.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66638' 2024-03-09T12:55:33.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.f 2024-03-09T12:55:33.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 18 2024-03-09T12:55:33.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.12 2024-03-09T12:55:33.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66643"' 2024-03-09T12:55:33.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66643' 2024-03-09T12:55:33.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66637: /' 2024-03-09T12:55:33.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66641: /' 2024-03-09T12:55:33.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.10 2024-03-09T12:55:33.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66650: /' 2024-03-09T12:55:33.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 19 2024-03-09T12:55:33.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.13 2024-03-09T12:55:33.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66652"' 2024-03-09T12:55:33.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66652' 2024-03-09T12:55:33.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.11 2024-03-09T12:55:33.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 20 2024-03-09T12:55:33.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.14 2024-03-09T12:55:33.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66657"' 2024-03-09T12:55:33.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66657' 2024-03-09T12:55:33.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66655: /' 2024-03-09T12:55:33.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66662: /' 2024-03-09T12:55:33.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.13 2024-03-09T12:55:33.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.12 2024-03-09T12:55:33.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 21 2024-03-09T12:55:33.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.14 2024-03-09T12:55:33.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.15 2024-03-09T12:55:33.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66678"' 2024-03-09T12:55:33.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66678' 2024-03-09T12:55:33.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66672: /' 2024-03-09T12:55:33.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 22 2024-03-09T12:55:33.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.16 2024-03-09T12:55:33.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66684"' 2024-03-09T12:55:33.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66684' 2024-03-09T12:55:33.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.15 2024-03-09T12:55:33.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66683: /' 2024-03-09T12:55:33.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 23 2024-03-09T12:55:33.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.17 2024-03-09T12:55:33.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66697"' 2024-03-09T12:55:33.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66697' 2024-03-09T12:55:33.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66694: /' 2024-03-09T12:55:33.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.16 2024-03-09T12:55:33.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 24 2024-03-09T12:55:33.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.18 2024-03-09T12:55:33.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66716"' 2024-03-09T12:55:33.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66716' 2024-03-09T12:55:33.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.17 2024-03-09T12:55:33.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66722: /' 2024-03-09T12:55:33.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 25 2024-03-09T12:55:33.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.19 2024-03-09T12:55:33.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66730"' 2024-03-09T12:55:33.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66730' 2024-03-09T12:55:33.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66725: /' 2024-03-09T12:55:33.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.18 2024-03-09T12:55:33.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 26 2024-03-09T12:55:33.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1a 2024-03-09T12:55:33.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66751"' 2024-03-09T12:55:33.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66751' 2024-03-09T12:55:33.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 27 2024-03-09T12:55:33.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1b 2024-03-09T12:55:33.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66771"' 2024-03-09T12:55:33.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66771' 2024-03-09T12:55:33.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66761: /' 2024-03-09T12:55:33.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.19 2024-03-09T12:55:33.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1a 2024-03-09T12:55:33.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66770: /' 2024-03-09T12:55:33.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1b 2024-03-09T12:55:33.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 28 2024-03-09T12:55:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1c 2024-03-09T12:55:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66803"' 2024-03-09T12:55:33.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66803' 2024-03-09T12:55:33.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66776: /' 2024-03-09T12:55:33.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 29 2024-03-09T12:55:33.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1d 2024-03-09T12:55:33.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66820"' 2024-03-09T12:55:33.849 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66820' 2024-03-09T12:55:33.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66816: /' 2024-03-09T12:55:33.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1c 2024-03-09T12:55:33.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 30 2024-03-09T12:55:33.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1e 2024-03-09T12:55:33.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66837"' 2024-03-09T12:55:33.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66837' 2024-03-09T12:55:33.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:302: TEST_recovery_scrub_2: for pg in $(seq 0 $(expr $PGS - 1)) 2024-03-09T12:55:33.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66836: /' 2024-03-09T12:55:33.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1d 2024-03-09T12:55:33.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: printf %x 31 2024-03-09T12:55:33.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:304: TEST_recovery_scrub_2: run_in_background pids pg_scrub_mod 1.1f 2024-03-09T12:55:33.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T12:55:33.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T12:55:33.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 66845"' 2024-03-09T12:55:33.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 66845' 2024-03-09T12:55:33.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:306: TEST_recovery_scrub_2: wait_background_check pids 2024-03-09T12:55:33.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:210: wait_background_check: pids=' 66531 66536 66544 66550 66559 66564 66573 66576 66584 66589 66599 66603 66607 66611 66615 66624 66630 66638 66643 66652 66657 66678 66684 66697 66716 66730 66751 66771 66803 66820 66837 66845' 2024-03-09T12:55:33.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:212: wait_background_check: return_code=0 2024-03-09T12:55:33.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:33.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66531 2024-03-09T12:55:33.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66844: /' 2024-03-09T12:55:33.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1e 2024-03-09T12:55:33.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/66850: /' 2024-03-09T12:55:33.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: pg_scrub_mod 1.1f 2024-03-09T12:55:40.691 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.0 2024-03-09T12:55:40.691 INFO:tasks.workunit.client.0.smithi162.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.0 2024-03-09T12:55:40.691 INFO:tasks.workunit.client.0.smithi162.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T12:55:40.691 INFO:tasks.workunit.client.0.smithi162.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:40.691 INFO:tasks.workunit.client.0.smithi162.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:40.691 INFO:tasks.workunit.client.0.smithi162.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T12:55:40.691 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:40.691 INFO:tasks.workunit.client.0.smithi162.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:40.692 INFO:tasks.workunit.client.0.smithi162.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.0 query 2024-03-09T12:55:40.692 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:40.692 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:40.692 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:40.692 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.0 2024-03-09T12:55:40.692 INFO:tasks.workunit.client.0.smithi162.stderr:66534: instructing pg 1.0 on osd.7 to scrub 2024-03-09T12:55:40.692 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:40.692 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.0") | .state' 2024-03-09T12:55:40.692 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:40.692 INFO:tasks.workunit.client.0.smithi162.stderr:66534: dumped pgs 2024-03-09T12:55:40.692 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.0 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:40.692 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.0 2024-03-09T12:55:40.692 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:66534: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:66534: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:40.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:40.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:40.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:40.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:40.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66536 2024-03-09T12:55:40.968 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1b 2024-03-09T12:55:40.968 INFO:tasks.workunit.client.0.smithi162.stderr:66776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1b 2024-03-09T12:55:40.968 INFO:tasks.workunit.client.0.smithi162.stderr:66776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-03-09T12:55:40.968 INFO:tasks.workunit.client.0.smithi162.stderr:66776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:40.968 INFO:tasks.workunit.client.0.smithi162.stderr:66776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:40.968 INFO:tasks.workunit.client.0.smithi162.stderr:66776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1b query 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1b 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: instructing pg 1.1b on osd.0 to scrub 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1b") | .state' 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: dumped pgs 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-03-09T12:55:40.969 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1b 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:40.970 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1b 2024-03-09T12:55:40.970 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:40.970 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:40.970 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:40.970 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:40.970 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:40.970 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:40.970 INFO:tasks.workunit.client.0.smithi162.stderr:66776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1b last_scrub_stamp 2024-03-09T12:55:40.970 INFO:tasks.workunit.client.0.smithi162.stderr:66776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1b 2024-03-09T12:55:40.970 INFO:tasks.workunit.client.0.smithi162.stderr:66776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:40.970 INFO:tasks.workunit.client.0.smithi162.stderr:66776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:41.009 INFO:tasks.workunit.client.0.smithi162.stderr:66776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stat66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.17 2024-03-09T12:55:41.010 INFO:tasks.workunit.client.0.smithi162.stderr:66725: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.17 2024-03-09T12:55:41.010 INFO:tasks.workunit.client.0.smithi162.stderr:66725: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-03-09T12:55:41.010 INFO:tasks.workunit.client.0.smithi162.stderr:66725: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:41.010 INFO:tasks.workunit.client.0.smithi162.stderr:66725: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-03-09T12:55:41.010 INFO:tasks.workunit.client.0.smithi162.stderr:66725: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:41.010 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.010 INFO:tasks.workunit.client.0.smithi162.stderr:66725: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:41.010 INFO:tasks.workunit.client.0.smithi162.stderr:66725: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.17 query 2024-03-09T12:55:41.010 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:41.010 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:41.010 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:41.010 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.17 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: instructing pg 1.17 on osd.0 to scrub 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.17") | .state' 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: dumped pgs 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.17 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.17 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:41.011 INFO:tasks.workunit.client.0.smithi162.stderr:66725: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.17 last_scrub_stamp 2024-03-09T12:55:41.012 INFO:tasks.workunit.client.0.smithi162.stderr:66725: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.17 2024-03-09T12:55:41.012 INFO:tasks.workunit.client.0.smithi162.stderr:66725: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:41.012 INFO:tasks.workunit.client.0.smithi162.stderr:66725: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:41.012 INFO:tasks.workunit.client.0.smithi162.stderr:66725: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.17") | .last_scrub_stamp' 2024-03-09T12:55:41.012 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:41.012 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:41.012 INFO:tasks.workunit.client.0.smithi162.stderr:66725: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:41.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:41.015 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.c 2024-03-09T12:55:41.015 INFO:tasks.workunit.client.0.smithi162.stderr:66614: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.c 2024-03-09T12:55:41.015 INFO:tasks.workunit.client.0.smithi162.stderr:66614: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-03-09T12:55:41.015 INFO:tasks.workunit.client.0.smithi162.stderr:66614: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:41.015 INFO:tasks.workunit.client.0.smithi162.stderr:66614: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-03-09T12:55:41.015 INFO:tasks.workunit.client.0.smithi162.stderr:66614: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.c query 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.c 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: instructing pg 1.c on osd.3 to scrub 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.c") | .state' 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: dumped pgs 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.c 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.016 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.c 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.c last_scrub_stamp 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.c 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.c") | .last_scrub_stamp' 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:66614: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:41.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:41.041 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.4 2024-03-09T12:55:41.041 INFO:tasks.workunit.client.0.smithi162.stderr:66561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.4 2024-03-09T12:55:41.042 INFO:tasks.workunit.client.0.smithi162.stderr:66561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-03-09T12:55:41.042 INFO:tasks.workunit.client.0.smithi162.stderr:66561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:41.042 INFO:tasks.workunit.client.0.smithi162.stderr:66561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-03-09T12:55:41.042 INFO:tasks.workunit.client.0.smithi162.stderr:66561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:41.042 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.042 INFO:tasks.workunit.client.0.smithi162.stderr:66561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:41.042 INFO:tasks.workunit.client.0.smithi162.stderr:66561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.4 query 2024-03-09T12:55:41.042 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:41.042 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:41.042 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:41.042 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.4 2024-03-09T12:55:41.042 INFO:tasks.workunit.client.0.smithi162.stderr:66561: instructing pg 1.4 on osd.3 to scrub 2024-03-09T12:55:41.042 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.4") | .state' 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: dumped pgs 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.4 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.4 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.4 last_scrub_stamp 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.4 2024-03-09T12:55:41.043 INFO:tasks.workunit.client.0.smithi162.stderr:66561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:41.044 INFO:tasks.workunit.client.0.smithi162.stderr:66561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:41.044 INFO:tasks.workunit.client.0.smithi162.stderr:66561: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2024-03-09T12:55:41.044 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:41.044 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:41.044 INFO:tasks.workunit.client.0.smithi162.stderr:66561: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:41.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.12 2024-03-09T12:55:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.12 2024-03-09T12:55:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-03-09T12:55:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-03-09T12:55:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.12 query 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.12 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: instructing pg 1.12 on osd.0 to scrub 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.12") | .state' 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: dumped pgs 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.12 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.12 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:41.480 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:41.480 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:41.480 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:41.480 INFO:tasks.workunit.client.0.smithi162.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.12 last_scrub_stamp 2024-03-09T12:55:41.480 INFO:tasks.workunit.client.0.smithi162.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.12 2024-03-09T12:55:41.480 INFO:tasks.workunit.client.0.smithi162.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:41.480 INFO:tasks.workunit.client.0.smithi162.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:41.480 INFO:tasks.workunit.client.0.smithi162.stderr:66655: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.12") | .last_scrub_stamp' 2024-03-09T12:55:41.480 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:41.481 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:41.481 INFO:tasks.workunit.client.0.smithi162.stderr:66655: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:41.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:41.487 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.d 2024-03-09T12:55:41.487 INFO:tasks.workunit.client.0.smithi162.stderr:66623: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.d 2024-03-09T12:55:41.487 INFO:tasks.workunit.client.0.smithi162.stderr:66623: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-03-09T12:55:41.487 INFO:tasks.workunit.client.0.smithi162.stderr:66623: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:41.487 INFO:tasks.workunit.client.0.smithi162.stderr:66623: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:41.488 INFO:tasks.workunit.client.0.smithi162.stderr:66623: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-03-09T12:55:41.488 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.488 INFO:tasks.workunit.client.0.smithi162.stderr:66623: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:41.488 INFO:tasks.workunit.client.0.smithi162.stderr:66623: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.d query 2024-03-09T12:55:41.488 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:41.488 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:41.488 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:41.489 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.d 2024-03-09T12:55:41.489 INFO:tasks.workunit.client.0.smithi162.stderr:66623: instructing pg 1.d on osd.7 to scrub 2024-03-09T12:55:41.489 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:41.489 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.d") | .state' 2024-03-09T12:55:41.489 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:41.489 INFO:tasks.workunit.client.0.smithi162.stderr:66623: dumped pgs 2024-03-09T12:55:41.489 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.d 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.489 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.d 2024-03-09T12:55:41.490 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.490 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:41.490 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:41.490 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:41.490 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:41.490 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:41.490 INFO:tasks.workunit.client.0.smithi162.stderr:66623: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.d last_scrub_stamp 2024-03-09T12:55:41.491 INFO:tasks.workunit.client.0.smithi162.stderr:66623: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.d 2024-03-09T12:55:41.491 INFO:tasks.workunit.client.0.smithi162.stderr:66623: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:41.491 INFO:tasks.workunit.client.0.smithi162.stderr:66623: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.d") | .last_scrub_stamp' 2024-03-09T12:55:41.491 INFO:tasks.workunit.client.0.smithi162.stderr:66623: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:41.491 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:41.491 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:41.491 INFO:tasks.workunit.client.0.smithi162.stderr:66623: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:41.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:41.859 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.7 2024-03-09T12:55:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.7 2024-03-09T12:55:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-03-09T12:55:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-03-09T12:55:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.862 INFO:tasks.workunit.client.0.smithi162.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.7 query 2024-03-09T12:55:41.862 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:41.862 INFO:tasks.workunit.client.0.smithi162.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:41.862 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:41.862 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:41.862 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.7 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: instructing pg 1.7 on osd.6 to scrub 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.7") | .state' 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: dumped pgs 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:198: pg_scrub_mod: recovery=true 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.7 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.7 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:41.863 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:41.864 INFO:tasks.workunit.client.0.smithi162.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.7 last_scrub_stamp 2024-03-09T12:55:41.864 INFO:tasks.workunit.client.0.smithi162.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.7 2024-03-09T12:55:41.864 INFO:tasks.workunit.client.0.smithi162.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:41.864 INFO:tasks.workunit.client.0.smithi162.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2024-03-09T12:55:42.062 INFO:tasks.workunit.client.0.smithi162.stderr:66580: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.9 2024-03-09T12:55:42.062 INFO:tasks.workunit.client.0.smithi162.stderr:66594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.9 2024-03-09T12:55:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:66594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-03-09T12:55:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:66594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:66594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:66594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-03-09T12:55:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:66594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.9 query 2024-03-09T12:55:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:66594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.9 2024-03-09T12:55:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:66594: instructing pg 1.9 on osd.1 to scrub 2024-03-09T12:55:42.063 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.9") | .state' 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: dumped pgs 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.9 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.9 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.9 last_scrub_stamp 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.9 2024-03-09T12:55:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:66594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.065 INFO:tasks.workunit.client.0.smithi162.stderr:66594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.065 INFO:tasks.workunit.client.0.smithi162.stderr:66594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.9") | .last_scrub_stamp' 2024-03-09T12:55:42.065 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.065 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.065 INFO:tasks.workunit.client.0.smithi162.stderr:66594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.132 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1 query 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: instructing pg 1.1 on osd.5 to scrub 2024-03-09T12:55:42.133 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1") | .state' 2024-03-09T12:55:42.134 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.134 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.134 INFO:tasks.workunit.client.0.smithi162.stderr:66541: dumped pgs 2024-03-09T12:55:42.134 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.134 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1 2024-03-09T12:55:42.134 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.135 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.135 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.135 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.135 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.135 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.135 INFO:tasks.workunit.client.0.smithi162.stderr:66541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1 last_scrub_stamp 2024-03-09T12:55:42.136 INFO:tasks.workunit.client.0.smithi162.stderr:66541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1 2024-03-09T12:55:42.136 INFO:tasks.workunit.client.0.smithi162.stderr:66541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.136 INFO:tasks.workunit.client.0.smithi162.stderr:66541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.136 INFO:tasks.workunit.client.0.smithi162.stderr:66541: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2024-03-09T12:55:42.136 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.136 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.137 INFO:tasks.workunit.client.0.smithi162.stderr:66541: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:42.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:42.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:42.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:42.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66544 2024-03-09T12:55:42.145 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.16 2024-03-09T12:55:42.145 INFO:tasks.workunit.client.0.smithi162.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.16 2024-03-09T12:55:42.145 INFO:tasks.workunit.client.0.smithi162.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-03-09T12:55:42.145 INFO:tasks.workunit.client.0.smithi162.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.145 INFO:tasks.workunit.client.0.smithi162.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.146 INFO:tasks.workunit.client.0.smithi162.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-03-09T12:55:42.146 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.146 INFO:tasks.workunit.client.0.smithi162.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.146 INFO:tasks.workunit.client.0.smithi162.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.16 query 2024-03-09T12:55:42.146 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.146 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.146 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.146 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.16 2024-03-09T12:55:42.146 INFO:tasks.workunit.client.0.smithi162.stderr:66694: instructing pg 1.16 on osd.6 to scrub 2024-03-09T12:55:42.147 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.16") | .state' 2024-03-09T12:55:42.147 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.147 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.147 INFO:tasks.workunit.client.0.smithi162.stderr:66694: dumped pgs 2024-03-09T12:55:42.147 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.16 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.147 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.16 2024-03-09T12:55:42.147 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.147 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.147 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.147 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.148 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.148 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.148 INFO:tasks.workunit.client.0.smithi162.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.16 last_scrub_stamp 2024-03-09T12:55:42.148 INFO:tasks.workunit.client.0.smithi162.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.16 2024-03-09T12:55:42.148 INFO:tasks.workunit.client.0.smithi162.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.148 INFO:tasks.workunit.client.0.smithi162.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.148 INFO:tasks.workunit.client.0.smithi162.stderr:66694: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.16") | .last_scrub_stamp' 2024-03-09T12:55:42.148 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.148 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.148 INFO:tasks.workunit.client.0.smithi162.stderr:66694: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.6 2024-03-09T12:55:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:66577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.6 2024-03-09T12:55:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:66577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-03-09T12:55:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:66577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:66577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-03-09T12:55:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:66577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:66577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:66577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.6 query 2024-03-09T12:55:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.6 2024-03-09T12:55:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:66577: instructing pg 1.6 on osd.6 to scrub 2024-03-09T12:55:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.6") | .state' 2024-03-09T12:55:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:66577: dumped pgs 2024-03-09T12:55:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.6 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.6 2024-03-09T12:55:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:66577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.6 last_scrub_stamp 2024-03-09T12:55:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:66577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.6 2024-03-09T12:55:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:66577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:66577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:66577: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2024-03-09T12:55:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:66577: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1e 2024-03-09T12:55:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:66844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1e 2024-03-09T12:55:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:66844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-03-09T12:55:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:66844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:66844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:66844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-03-09T12:55:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:66844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:66844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1e query 2024-03-09T12:55:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.334 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1e 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: instructing pg 1.1e on osd.3 to scrub 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1e") | .state' 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: dumped pgs 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1e 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1e 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:66844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1e last_scrub_stamp 2024-03-09T12:55:42.336 INFO:tasks.workunit.client.0.smithi162.stderr:66844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1e 2024-03-09T12:55:42.336 INFO:tasks.workunit.client.0.smithi162.stderr:66844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.336 INFO:tasks.workunit.client.0.smithi162.stderr:66844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.336 INFO:tasks.workunit.client.0.smithi162.stderr:66844: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1e") | .last_scrub_stamp' 2024-03-09T12:55:42.336 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.336 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.336 INFO:tasks.workunit.client.0.smithi162.stderr:66844: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.416 INFO:tasks.workunit.client.0.smithi162.stderr:s | .[] | select(.pgid=="1.1b") | .last_scrub_stamp' 2024-03-09T12:55:42.417 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.417 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.417 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-03-09T12:55:42.417 INFO:tasks.workunit.client.0.smithi162.stderr:66776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-03-09T12:55:42.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-03-09T12:55:42.435 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.2 2024-03-09T12:55:42.435 INFO:tasks.workunit.client.0.smithi162.stderr:66548: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.2 2024-03-09T12:55:42.435 INFO:tasks.workunit.client.0.smithi162.stderr:66548: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-03-09T12:55:42.436 INFO:tasks.workunit.client.0.smithi162.stderr:66548: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.436 INFO:tasks.workunit.client.0.smithi162.stderr:66548: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.436 INFO:tasks.workunit.client.0.smithi162.stderr:66548: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-03-09T12:55:42.436 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.436 INFO:tasks.workunit.client.0.smithi162.stderr:66548: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.2 query 2024-03-09T12:55:42.436 INFO:tasks.workunit.client.0.smithi162.stderr:66548: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.436 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.436 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.436 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.436 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.2 2024-03-09T12:55:42.436 INFO:tasks.workunit.client.0.smithi162.stderr:66548: instructing pg 1.2 on osd.0 to scrub 2024-03-09T12:55:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.2") | .state' 2024-03-09T12:55:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:66548: dumped pgs 2024-03-09T12:55:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.2 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.2 2024-03-09T12:55:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.437 INFO:tasks.workunit.client.0.smithi162.stderr:66548: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.2 last_scrub_stamp 2024-03-09T12:55:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:66548: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.2 2024-03-09T12:55:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:66548: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:66548: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:66548: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2024-03-09T12:55:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:66548: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:42.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66550 2024-03-09T12:55:42.472 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.8 2024-03-09T12:55:42.472 INFO:tasks.workunit.client.0.smithi162.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.8 2024-03-09T12:55:42.472 INFO:tasks.workunit.client.0.smithi162.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-03-09T12:55:42.472 INFO:tasks.workunit.client.0.smithi162.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.472 INFO:tasks.workunit.client.0.smithi162.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.472 INFO:tasks.workunit.client.0.smithi162.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-03-09T12:55:42.472 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.472 INFO:tasks.workunit.client.0.smithi162.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.8 query 2024-03-09T12:55:42.472 INFO:tasks.workunit.client.0.smithi162.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.472 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.472 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.8 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: instructing pg 1.8 on osd.5 to scrub 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.8") | .state' 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: dumped pgs 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.8 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.8 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.473 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.8 last_scrub_stamp 2024-03-09T12:55:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.8 2024-03-09T12:55:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:66587: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.8") | .last_scrub_stamp' 2024-03-09T12:55:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:66587: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.3 2024-03-09T12:55:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:66553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.3 2024-03-09T12:55:42.474 INFO:tasks.workunit.client.0.smithi162.stderr:66553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.3 query 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.3 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: instructing pg 1.3 on osd.4 to scrub 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.3") | .state' 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.475 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: dumped pgs 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.3 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.3 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.476 INFO:tasks.workunit.client.0.smithi162.stderr:66553: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:66553: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66559 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:42.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66564 2024-03-09T12:55:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1d 2024-03-09T12:55:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:66836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1d 2024-03-09T12:55:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:66836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-03-09T12:55:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:66836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:66836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-03-09T12:55:42.478 INFO:tasks.workunit.client.0.smithi162.stderr:66836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1d query 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1d 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: instructing pg 1.1d on osd.1 to scrub 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1d") | .state' 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: dumped pgs 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1d 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.479 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1d 2024-03-09T12:55:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:66836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1d last_scrub_stamp 2024-03-09T12:55:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:66836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1d 2024-03-09T12:55:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:66836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:66836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:66836: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1d") | .last_scrub_stamp' 2024-03-09T12:55:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.480 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.481 INFO:tasks.workunit.client.0.smithi162.stderr:66836: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.512 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.a 2024-03-09T12:55:42.513 INFO:tasks.workunit.client.0.smithi162.stderr:66606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.a 2024-03-09T12:55:42.513 INFO:tasks.workunit.client.0.smithi162.stderr:66606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-03-09T12:55:42.513 INFO:tasks.workunit.client.0.smithi162.stderr:66606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.513 INFO:tasks.workunit.client.0.smithi162.stderr:66606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-03-09T12:55:42.513 INFO:tasks.workunit.client.0.smithi162.stderr:66606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.513 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.513 INFO:tasks.workunit.client.0.smithi162.stderr:66606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.a query 2024-03-09T12:55:42.513 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.513 INFO:tasks.workunit.client.0.smithi162.stderr:66606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.513 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.513 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.513 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.a 2024-03-09T12:55:42.513 INFO:tasks.workunit.client.0.smithi162.stderr:66606: instructing pg 1.a on osd.1 to scrub 2024-03-09T12:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.a") | .state' 2024-03-09T12:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:66606: dumped pgs 2024-03-09T12:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.a 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.a 2024-03-09T12:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:66606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.a last_scrub_stamp 2024-03-09T12:55:42.515 INFO:tasks.workunit.client.0.smithi162.stderr:66606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.a 2024-03-09T12:55:42.515 INFO:tasks.workunit.client.0.smithi162.stderr:66606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.515 INFO:tasks.workunit.client.0.smithi162.stderr:66606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.515 INFO:tasks.workunit.client.0.smithi162.stderr:66606: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.a") | .last_scrub_stamp' 2024-03-09T12:55:42.515 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.515 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.515 INFO:tasks.workunit.client.0.smithi162.stderr:66606: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.541 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1f 2024-03-09T12:55:42.541 INFO:tasks.workunit.client.0.smithi162.stderr:66850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1f 2024-03-09T12:55:42.541 INFO:tasks.workunit.client.0.smithi162.stderr:66850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-03-09T12:55:42.541 INFO:tasks.workunit.client.0.smithi162.stderr:66850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:66850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-03-09T12:55:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:66850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:66850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1f query 2024-03-09T12:55:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:66850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1f 2024-03-09T12:55:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:66850: instructing pg 1.1f on osd.5 to scrub 2024-03-09T12:55:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1f") | .state' 2024-03-09T12:55:42.542 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:66850: dumped pgs 2024-03-09T12:55:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1f 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1f 2024-03-09T12:55:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:66850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1f last_scrub_stamp 2024-03-09T12:55:42.543 INFO:tasks.workunit.client.0.smithi162.stderr:66850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1f 2024-03-09T12:55:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:66850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:66850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:66850: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1f") | .last_scrub_stamp' 2024-03-09T12:55:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:66850: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.691 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.e 2024-03-09T12:55:42.692 INFO:tasks.workunit.client.0.smithi162.stderr:66621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.e 2024-03-09T12:55:42.692 INFO:tasks.workunit.client.0.smithi162.stderr:66621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-03-09T12:55:42.692 INFO:tasks.workunit.client.0.smithi162.stderr:66621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.692 INFO:tasks.workunit.client.0.smithi162.stderr:66621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.692 INFO:tasks.workunit.client.0.smithi162.stderr:66621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-03-09T12:55:42.692 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.692 INFO:tasks.workunit.client.0.smithi162.stderr:66621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.e query 2024-03-09T12:55:42.692 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.692 INFO:tasks.workunit.client.0.smithi162.stderr:66621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.692 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.692 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.692 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.e 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: instructing pg 1.e on osd.1 to scrub 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.e") | .state' 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: dumped pgs 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.e 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.e 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.693 INFO:tasks.workunit.client.0.smithi162.stderr:66621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.e last_scrub_stamp 2024-03-09T12:55:42.694 INFO:tasks.workunit.client.0.smithi162.stderr:66621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.e 2024-03-09T12:55:42.694 INFO:tasks.workunit.client.0.smithi162.stderr:66621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.694 INFO:tasks.workunit.client.0.smithi162.stderr:66621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.e") | .last_scrub_stamp' 2024-03-09T12:55:42.694 INFO:tasks.workunit.client.0.smithi162.stderr:66621: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.694 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.694 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.694 INFO:tasks.workunit.client.0.smithi162.stderr:66621: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.695 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.10 2024-03-09T12:55:42.695 INFO:tasks.workunit.client.0.smithi162.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.10 2024-03-09T12:55:42.695 INFO:tasks.workunit.client.0.smithi162.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-03-09T12:55:42.695 INFO:tasks.workunit.client.0.smithi162.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.695 INFO:tasks.workunit.client.0.smithi162.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.695 INFO:tasks.workunit.client.0.smithi162.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-03-09T12:55:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.10 query 2024-03-09T12:55:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.10 2024-03-09T12:55:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:66641: instructing pg 1.10 on osd.5 to scrub 2024-03-09T12:55:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.10") | .state' 2024-03-09T12:55:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:66641: dumped pgs 2024-03-09T12:55:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.10 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.10 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.10 last_scrub_stamp 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.10 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.10") | .last_scrub_stamp' 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.697 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.698 INFO:tasks.workunit.client.0.smithi162.stderr:66641: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.753 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.13 2024-03-09T12:55:42.754 INFO:tasks.workunit.client.0.smithi162.stderr:66662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.13 2024-03-09T12:55:42.754 INFO:tasks.workunit.client.0.smithi162.stderr:66662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-03-09T12:55:42.754 INFO:tasks.workunit.client.0.smithi162.stderr:66662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.754 INFO:tasks.workunit.client.0.smithi162.stderr:66662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-03-09T12:55:42.754 INFO:tasks.workunit.client.0.smithi162.stderr:66662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.754 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.754 INFO:tasks.workunit.client.0.smithi162.stderr:66662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.13 query 2024-03-09T12:55:42.754 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.754 INFO:tasks.workunit.client.0.smithi162.stderr:66662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.754 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.754 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.754 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.13 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: instructing pg 1.13 on osd.4 to scrub 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.13") | .state' 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: dumped pgs 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.13 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.13 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.13 last_scrub_stamp 2024-03-09T12:55:42.755 INFO:tasks.workunit.client.0.smithi162.stderr:66662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.13 2024-03-09T12:55:42.756 INFO:tasks.workunit.client.0.smithi162.stderr:66662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.756 INFO:tasks.workunit.client.0.smithi162.stderr:66662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.756 INFO:tasks.workunit.client.0.smithi162.stderr:66662: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.13") | .last_scrub_stamp' 2024-03-09T12:55:42.756 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.756 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.756 INFO:tasks.workunit.client.0.smithi162.stderr:66662: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.774 INFO:tasks.workunit.client.0.smithi162.stderr:_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.774 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.774 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.774 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test true = true 2024-03-09T12:55:42.774 INFO:tasks.workunit.client.0.smithi162.stderr:66580: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:203: pg_scrub_mod: return 2 2024-03-09T12:55:42.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 2 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.18 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.18 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.18 query 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.18 2024-03-09T12:55:42.775 INFO:tasks.workunit.client.0.smithi162.stderr:66722: instructing pg 1.18 on osd.5 to scrub 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.18") | .state' 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: dumped pgs 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.18 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.18 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.18 last_scrub_stamp 2024-03-09T12:55:42.776 INFO:tasks.workunit.client.0.smithi162.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.18 2024-03-09T12:55:42.777 INFO:tasks.workunit.client.0.smithi162.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.777 INFO:tasks.workunit.client.0.smithi162.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.777 INFO:tasks.workunit.client.0.smithi162.stderr:66722: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.18") | .last_scrub_stamp' 2024-03-09T12:55:42.777 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.777 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.777 INFO:tasks.workunit.client.0.smithi162.stderr:66722: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.870 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.11 2024-03-09T12:55:42.870 INFO:tasks.workunit.client.0.smithi162.stderr:66650: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.11 2024-03-09T12:55:42.870 INFO:tasks.workunit.client.0.smithi162.stderr:66650: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-03-09T12:55:42.870 INFO:tasks.workunit.client.0.smithi162.stderr:66650: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.870 INFO:tasks.workunit.client.0.smithi162.stderr:66650: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-03-09T12:55:42.870 INFO:tasks.workunit.client.0.smithi162.stderr:66650: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:66650: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:66650: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.11 query 2024-03-09T12:55:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.11 2024-03-09T12:55:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:66650: instructing pg 1.11 on osd.0 to scrub 2024-03-09T12:55:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.11") | .state' 2024-03-09T12:55:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:66650: dumped pgs 2024-03-09T12:55:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.11 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.11 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.11 last_scrub_stamp 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.11 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.11") | .last_scrub_stamp' 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.872 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.873 INFO:tasks.workunit.client.0.smithi162.stderr:66650: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.907 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.f 2024-03-09T12:55:42.907 INFO:tasks.workunit.client.0.smithi162.stderr:66637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.f 2024-03-09T12:55:42.907 INFO:tasks.workunit.client.0.smithi162.stderr:66637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-03-09T12:55:42.907 INFO:tasks.workunit.client.0.smithi162.stderr:66637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.907 INFO:tasks.workunit.client.0.smithi162.stderr:66637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-03-09T12:55:42.907 INFO:tasks.workunit.client.0.smithi162.stderr:66637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.907 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.907 INFO:tasks.workunit.client.0.smithi162.stderr:66637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.908 INFO:tasks.workunit.client.0.smithi162.stderr:66637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.f query 2024-03-09T12:55:42.908 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.908 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.908 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.908 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.f 2024-03-09T12:55:42.908 INFO:tasks.workunit.client.0.smithi162.stderr:66637: instructing pg 1.f on osd.6 to scrub 2024-03-09T12:55:42.908 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.908 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.908 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.f") | .state' 2024-03-09T12:55:42.908 INFO:tasks.workunit.client.0.smithi162.stderr:66637: dumped pgs 2024-03-09T12:55:42.908 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.f 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.908 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.f 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.f last_scrub_stamp 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.f 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.f") | .last_scrub_stamp' 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:66637: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.929 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.b 2024-03-09T12:55:42.929 INFO:tasks.workunit.client.0.smithi162.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.b 2024-03-09T12:55:42.929 INFO:tasks.workunit.client.0.smithi162.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-03-09T12:55:42.929 INFO:tasks.workunit.client.0.smithi162.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.929 INFO:tasks.workunit.client.0.smithi162.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-03-09T12:55:42.929 INFO:tasks.workunit.client.0.smithi162.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.929 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.929 INFO:tasks.workunit.client.0.smithi162.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.929 INFO:tasks.workunit.client.0.smithi162.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.b query 2024-03-09T12:55:42.929 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.929 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.929 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.930 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.b 2024-03-09T12:55:42.930 INFO:tasks.workunit.client.0.smithi162.stderr:66610: instructing pg 1.b on osd.1 to scrub 2024-03-09T12:55:42.930 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.930 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.b") | .state' 2024-03-09T12:55:42.930 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.930 INFO:tasks.workunit.client.0.smithi162.stderr:66610: dumped pgs 2024-03-09T12:55:42.930 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.b 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.930 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.b 2024-03-09T12:55:42.930 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.930 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.930 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.930 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.930 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.931 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.931 INFO:tasks.workunit.client.0.smithi162.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.b last_scrub_stamp 2024-03-09T12:55:42.931 INFO:tasks.workunit.client.0.smithi162.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.b 2024-03-09T12:55:42.931 INFO:tasks.workunit.client.0.smithi162.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.931 INFO:tasks.workunit.client.0.smithi162.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.931 INFO:tasks.workunit.client.0.smithi162.stderr:66610: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.b") | .last_scrub_stamp' 2024-03-09T12:55:42.931 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.931 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.931 INFO:tasks.workunit.client.0.smithi162.stderr:66610: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.943 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1c 2024-03-09T12:55:42.943 INFO:tasks.workunit.client.0.smithi162.stderr:66816: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1c 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1c query 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1c 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: instructing pg 1.1c on osd.0 to scrub 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1c") | .state' 2024-03-09T12:55:42.944 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: dumped pgs 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1c 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1c 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1c last_scrub_stamp 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1c 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.945 INFO:tasks.workunit.client.0.smithi162.stderr:66816: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.946 INFO:tasks.workunit.client.0.smithi162.stderr:66816: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1c") | .last_scrub_stamp' 2024-03-09T12:55:42.946 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.946 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.946 INFO:tasks.workunit.client.0.smithi162.stderr:66816: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:42.995 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.14 2024-03-09T12:55:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.14 2024-03-09T12:55:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-03-09T12:55:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-03-09T12:55:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.14 query 2024-03-09T12:55:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.14 2024-03-09T12:55:42.996 INFO:tasks.workunit.client.0.smithi162.stderr:66672: instructing pg 1.14 on osd.2 to scrub 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.14") | .state' 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: dumped pgs 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.14 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.14 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.14 last_scrub_stamp 2024-03-09T12:55:42.997 INFO:tasks.workunit.client.0.smithi162.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.14 2024-03-09T12:55:42.998 INFO:tasks.workunit.client.0.smithi162.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:42.998 INFO:tasks.workunit.client.0.smithi162.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.14") | .last_scrub_stamp' 2024-03-09T12:55:42.998 INFO:tasks.workunit.client.0.smithi162.stderr:66672: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:42.998 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:42.998 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:42.998 INFO:tasks.workunit.client.0.smithi162.stderr:66672: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:42.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.5 2024-03-09T12:55:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:66569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.5 2024-03-09T12:55:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:66569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-03-09T12:55:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:66569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:66569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:66569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-03-09T12:55:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:66569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:66569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.5 query 2024-03-09T12:55:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.5 2024-03-09T12:55:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:66569: instructing pg 1.5 on osd.4 to scrub 2024-03-09T12:55:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.5") | .state' 2024-03-09T12:55:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:66569: dumped pgs 2024-03-09T12:55:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.5 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.5 2024-03-09T12:55:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:43.110 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:66569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.5 last_scrub_stamp 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:66569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.5 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:66569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:66569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:66569: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:66569: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66573 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66576 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 0 + 1 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=1 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66584 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66589 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66599 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66603 2024-03-09T12:55:43.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66607 2024-03-09T12:55:43.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66611 2024-03-09T12:55:43.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66615 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66624 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66630 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66638 2024-03-09T12:55:43.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66643 2024-03-09T12:55:43.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66652 2024-03-09T12:55:43.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66657 2024-03-09T12:55:43.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66678 2024-03-09T12:55:43.118 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.19 2024-03-09T12:55:43.118 INFO:tasks.workunit.client.0.smithi162.stderr:66761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.19 2024-03-09T12:55:43.118 INFO:tasks.workunit.client.0.smithi162.stderr:66761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-03-09T12:55:43.118 INFO:tasks.workunit.client.0.smithi162.stderr:66761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:43.118 INFO:tasks.workunit.client.0.smithi162.stderr:66761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:43.118 INFO:tasks.workunit.client.0.smithi162.stderr:66761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.19 query 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.19 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: instructing pg 1.19 on osd.3 to scrub 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.19") | .state' 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: dumped pgs 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.19 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.19 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.19 last_scrub_stamp 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.19 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.19") | .last_scrub_stamp' 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:66761: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:43.209 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.15 2024-03-09T12:55:43.209 INFO:tasks.workunit.client.0.smithi162.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.15 2024-03-09T12:55:43.209 INFO:tasks.workunit.client.0.smithi162.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-03-09T12:55:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-03-09T12:55:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.15 query 2024-03-09T12:55:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.15 2024-03-09T12:55:43.210 INFO:tasks.workunit.client.0.smithi162.stderr:66683: instructing pg 1.15 on osd.1 to scrub 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.15") | .state' 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: dumped pgs 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.15 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.15 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.15 last_scrub_stamp 2024-03-09T12:55:43.211 INFO:tasks.workunit.client.0.smithi162.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.15 2024-03-09T12:55:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.15") | .last_scrub_stamp' 2024-03-09T12:55:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:66683: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:54:15.378828+0000 '>' '' 2024-03-09T12:55:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:66683: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66684 2024-03-09T12:55:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66697 2024-03-09T12:55:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66716 2024-03-09T12:55:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66730 2024-03-09T12:55:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66751 2024-03-09T12:55:43.232 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:189: pg_scrub_mod: local pgid=1.1a 2024-03-09T12:55:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:66770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: get_last_scrub_stamp 1.1a 2024-03-09T12:55:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:66770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-03-09T12:55:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:66770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:66770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:66770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-03-09T12:55:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:190: pg_scrub_mod: local last_scrub=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:66770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: jq '.acting[0]' 2024-03-09T12:55:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:66770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: bin/ceph pg 1.1a query 2024-03-09T12:55:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh: line 192: bin/ceph: No such file or directory 2024-03-09T12:55:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:192: pg_scrub_mod: local my_primary= 2024-03-09T12:55:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:193: pg_scrub_mod: local recovery=false 2024-03-09T12:55:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:194: pg_scrub_mod: ceph pg scrub 1.1a 2024-03-09T12:55:43.233 INFO:tasks.workunit.client.0.smithi162.stderr:66770: instructing pg 1.1a on osd.1 to scrub 2024-03-09T12:55:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: ceph --format json pg dump pgs 2024-03-09T12:55:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: jq '.pg_stats | .[] | select(.pgid == "1.1a") | .state' 2024-03-09T12:55:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:196: pg_scrub_mod: grep -q recovering 2024-03-09T12:55:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:66770: dumped pgs 2024-03-09T12:55:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:200: pg_scrub_mod: wait_for_scrub_mod 1.1a 2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:154: wait_for_scrub_mod: local pgid=1.1a 2024-03-09T12:55:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:155: wait_for_scrub_mod: local orig_primary=2024-03-09T12:54:15.378828+0000 2024-03-09T12:55:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:156: wait_for_scrub_mod: local last_scrub= 2024-03-09T12:55:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:157: wait_for_scrub_mod: local sname=last_scrub_stamp 2024-03-09T12:55:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i=0 )) 2024-03-09T12:55:43.234 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:159: wait_for_scrub_mod: (( i < 300 )) 2024-03-09T12:55:43.235 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:160: wait_for_scrub_mod: sleep 0.2 2024-03-09T12:55:43.235 INFO:tasks.workunit.client.0.smithi162.stderr:66770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: get_last_scrub_stamp 1.1a last_scrub_stamp 2024-03-09T12:55:43.235 INFO:tasks.workunit.client.0.smithi162.stderr:66770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.1a 2024-03-09T12:55:43.235 INFO:tasks.workunit.client.0.smithi162.stderr:66770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:55:43.235 INFO:tasks.workunit.client.0.smithi162.stderr:66770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1a") | .last_scrub_stamp' 2024-03-09T12:55:43.235 INFO:tasks.workunit.client.0.smithi162.stderr:66770: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:55:43.235 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:161: wait_for_scrub_mod: test 2024-03-09T12:55:39.888989+0000 '>' '' 2024-03-09T12:55:43.235 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:162: wait_for_scrub_mod: return 0 2024-03-09T12:55:43.235 INFO:tasks.workunit.client.0.smithi162.stderr:66770: /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:201: pg_scrub_mod: test false = true 2024-03-09T12:55:43.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T12:55:43.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66771 2024-03-09T12:55:43.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=2 2024-03-09T12:55:43.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 2 -eq 2 2024-03-09T12:55:43.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: expr 1 + 1 2024-03-09T12:55:43.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:218: wait_background_check: recov_scrub_count=2 2024-03-09T12:55:43.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66803 2024-03-09T12:55:43.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66820 2024-03-09T12:55:43.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66837 2024-03-09T12:55:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:213: wait_background_check: for pid in $pids 2024-03-09T12:55:43.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:214: wait_background_check: wait 66845 2024-03-09T12:55:43.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:215: wait_background_check: retcode=0 2024-03-09T12:55:43.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:216: wait_background_check: test 0 -eq 2 2024-03-09T12:55:43.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:219: wait_background_check: test 0 -ne 0 2024-03-09T12:55:43.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: eval 'pids='\'''\''' 2024-03-09T12:55:43.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:227: wait_background_check: pids= 2024-03-09T12:55:43.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:229: wait_background_check: return 0 2024-03-09T12:55:43.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:307: TEST_recovery_scrub_2: return_code=0 2024-03-09T12:55:43.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:308: TEST_recovery_scrub_2: '[' 0 -ne 0 ']' 2024-03-09T12:55:43.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:310: TEST_recovery_scrub_2: ERRORS=0 2024-03-09T12:55:43.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:311: TEST_recovery_scrub_2: test 2 -eq 0 2024-03-09T12:55:43.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: find td/osd-recovery-scrub 2024-03-09T12:55:43.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: grep '[^/]*\.pid' 2024-03-09T12:55:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:317: TEST_recovery_scrub_2: pidfile='td/osd-recovery-scrub/mon.a.pid 2024-03-09T12:55:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.5.pid 2024-03-09T12:55:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.6.pid 2024-03-09T12:55:43.248 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.1.pid 2024-03-09T12:55:43.249 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/mgr.x.pid 2024-03-09T12:55:43.249 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.0.pid 2024-03-09T12:55:43.249 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.4.pid 2024-03-09T12:55:43.249 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.3.pid 2024-03-09T12:55:43.249 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.7.pid 2024-03-09T12:55:43.250 INFO:tasks.workunit.client.0.smithi162.stderr:td/osd-recovery-scrub/osd.2.pid' 2024-03-09T12:55:43.250 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: cat td/osd-recovery-scrub/mon.a.pid td/osd-recovery-scrub/osd.5.pid td/osd-recovery-scrub/osd.6.pid td/osd-recovery-scrub/osd.1.pid td/osd-recovery-scrub/mgr.x.pid td/osd-recovery-scrub/osd.0.pid td/osd-recovery-scrub/osd.4.pid td/osd-recovery-scrub/osd.3.pid td/osd-recovery-scrub/osd.7.pid td/osd-recovery-scrub/osd.2.pid 2024-03-09T12:55:43.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:318: TEST_recovery_scrub_2: pid='53588 2024-03-09T12:55:43.250 INFO:tasks.workunit.client.0.smithi162.stderr:61498 2024-03-09T12:55:43.250 INFO:tasks.workunit.client.0.smithi162.stderr:62889 2024-03-09T12:55:43.250 INFO:tasks.workunit.client.0.smithi162.stderr:55994 2024-03-09T12:55:43.251 INFO:tasks.workunit.client.0.smithi162.stderr:53672 2024-03-09T12:55:43.251 INFO:tasks.workunit.client.0.smithi162.stderr:54618 2024-03-09T12:55:43.251 INFO:tasks.workunit.client.0.smithi162.stderr:60122 2024-03-09T12:55:43.251 INFO:tasks.workunit.client.0.smithi162.stderr:58746 2024-03-09T12:55:43.251 INFO:tasks.workunit.client.0.smithi162.stderr:64266 2024-03-09T12:55:43.251 INFO:tasks.workunit.client.0.smithi162.stderr:57370' 2024-03-09T12:55:43.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:319: TEST_recovery_scrub_2: kill -0 53588 61498 62889 55994 53672 54618 60122 58746 64266 57370 2024-03-09T12:55:43.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:327: TEST_recovery_scrub_2: kill_daemons td/osd-recovery-scrub 2024-03-09T12:55:43.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:55:43.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:55:43.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:55:43.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:55:43.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:55:43.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:55:43.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:329: TEST_recovery_scrub_2: declare -a err_strings 2024-03-09T12:55:43.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:330: TEST_recovery_scrub_2: err_strings[0]='not scheduling scrubs due to active recovery' 2024-03-09T12:55:43.548 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: expr 8 - 1 2024-03-09T12:55:43.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: seq 0 7 2024-03-09T12:55:43.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.0.log 2024-03-09T12:55:43.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.1.log 2024-03-09T12:55:43.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.2.log 2024-03-09T12:55:43.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.3.log 2024-03-09T12:55:43.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.4.log 2024-03-09T12:55:43.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.5.log 2024-03-09T12:55:43.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.6.log 2024-03-09T12:55:43.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:332: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:334: TEST_recovery_scrub_2: grep 'not scheduling scrubs' td/osd-recovery-scrub/osd.7.log 2024-03-09T12:55:43.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:336: TEST_recovery_scrub_2: for err_string in "${err_strings[@]}" 2024-03-09T12:55:43.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:338: TEST_recovery_scrub_2: found=false 2024-03-09T12:55:43.577 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: expr 8 - 1 2024-03-09T12:55:43.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: seq 0 7 2024-03-09T12:55:43.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.0.log 2024-03-09T12:55:43.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.1.log 2024-03-09T12:55:43.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.2.log 2024-03-09T12:55:43.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.3.log 2024-03-09T12:55:43.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.4.log 2024-03-09T12:55:43.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.5.log 2024-03-09T12:55:43.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.6.log 2024-03-09T12:55:43.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:339: TEST_recovery_scrub_2: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T12:55:43.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:341: TEST_recovery_scrub_2: grep 'not scheduling scrubs due to active recovery' td/osd-recovery-scrub/osd.7.log 2024-03-09T12:55:43.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:346: TEST_recovery_scrub_2: '[' false = true ']' 2024-03-09T12:55:43.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:352: TEST_recovery_scrub_2: teardown td/osd-recovery-scrub 2024-03-09T12:55:43.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-09T12:55:43.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T12:55:43.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-09T12:55:43.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:55:43.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:55:43.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:55:43.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:55:43.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:55:43.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:55:43.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T12:55:43.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T12:55:43.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T12:55:43.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T12:55:43.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T12:55:43.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T12:55:43.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:55:43.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T12:55:43.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T12:55:43.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:55:43.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T12:55:43.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T12:55:43.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-09T12:55:43.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T12:55:43.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:55:43.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:55:43.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44981 2024-03-09T12:55:43.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T12:55:43.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T12:55:43.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:354: TEST_recovery_scrub_2: '[' 0 '!=' 0 ']' 2024-03-09T12:55:43.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:360: TEST_recovery_scrub_2: echo 'TEST PASSED' 2024-03-09T12:55:43.639 INFO:tasks.workunit.client.0.smithi162.stdout:TEST PASSED 2024-03-09T12:55:43.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:361: TEST_recovery_scrub_2: return 0 2024-03-09T12:55:43.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-09T12:55:43.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-recovery-scrub 0 2024-03-09T12:55:43.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-scrub 2024-03-09T12:55:43.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-09T12:55:43.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-scrub KILL 2024-03-09T12:55:43.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:55:43.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:55:43.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:55:43.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:55:43.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:55:43.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:55:43.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T12:55:43.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T12:55:43.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T12:55:43.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T12:55:43.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T12:55:43.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T12:55:43.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:55:43.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T12:55:43.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T12:55:43.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:55:43.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T12:55:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-09T12:55:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-scrub 2024-03-09T12:55:43.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T12:55:43.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:55:43.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.44981 2024-03-09T12:55:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.44981 2024-03-09T12:55:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T12:55:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T12:55:43.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-09T12:55:43.649 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T12:55:43.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T12:55:43.679 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2024-03-09T12:55:43.679 DEBUG:teuthology.orchestra.run.smithi162:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6b01fe95f3aa686f1951a0c5828907a52930b92a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2024-03-09T12:55:43.740 INFO:tasks.workunit.client.0.smithi162.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-09T12:55:43.745 INFO:tasks.workunit.client.0.smithi162.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-09T12:55:43.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T12:55:43.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T12:55:43.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-09T12:55:43.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-09T12:55:43.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-09T12:55:43.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-09T12:55:43.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-09T12:55:43.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-09T12:55:43.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-dump 2024-03-09T12:55:43.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2024-03-09T12:55:43.746 INFO:tasks.workunit.client.0.smithi162.stdout:This test is disabled 2024-03-09T12:55:43.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2024-03-09T12:55:43.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-03-09T12:55:43.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-dump 0 2024-03-09T12:55:43.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-dump 2024-03-09T12:55:43.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-03-09T12:55:43.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-dump KILL 2024-03-09T12:55:43.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:55:43.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:55:43.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:55:43.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:55:43.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:55:43.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:55:43.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T12:55:43.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T12:55:43.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T12:55:43.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T12:55:43.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T12:55:43.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T12:55:43.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:55:43.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T12:55:43.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T12:55:43.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:55:43.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T12:55:43.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-03-09T12:55:43.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-dump 2024-03-09T12:55:43.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T12:55:43.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:55:43.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69987 2024-03-09T12:55:43.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69987 2024-03-09T12:55:43.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T12:55:43.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T12:55:43.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-03-09T12:55:43.755 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-09T12:55:43.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T12:55:43.823 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2024-03-09T12:55:43.823 DEBUG:teuthology.orchestra.run.smithi162:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6b01fe95f3aa686f1951a0c5828907a52930b92a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2024-03-09T12:55:43.884 INFO:tasks.workunit.client.0.smithi162.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-03-09T12:55:43.884 INFO:tasks.workunit.client.0.smithi162.stderr:++ TIMEOUT=300 2024-03-09T12:55:43.884 INFO:tasks.workunit.client.0.smithi162.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:++ MAX_TIMEOUT=15 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:++ PG_NUM=4 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:++ TMPDIR=/tmp 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:++ type xmlstarlet 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:++ XMLSTARLET=xmlstarlet 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:+++ uname 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' Linux = FreeBSD ']' 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:++ SED=sed 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:++ AWK=awk 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:+++ stty -a 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:+++ head -1 2024-03-09T12:55:43.885 INFO:tasks.workunit.client.0.smithi162.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-03-09T12:55:43.886 INFO:tasks.workunit.client.0.smithi162.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-03-09T12:55:43.887 INFO:tasks.workunit.client.0.smithi162.stderr:++ termwidth= 2024-03-09T12:55:43.887 INFO:tasks.workunit.client.0.smithi162.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-03-09T12:55:43.887 INFO:tasks.workunit.client.0.smithi162.stderr:++ DIFFCOLOPTS='-y ' 2024-03-09T12:55:43.887 INFO:tasks.workunit.client.0.smithi162.stderr:++ KERNCORE=kernel.core_pattern 2024-03-09T12:55:43.887 INFO:tasks.workunit.client.0.smithi162.stderr:++ EXTRA_OPTS= 2024-03-09T12:55:43.888 INFO:tasks.workunit.client.0.smithi162.stderr:++ test '' = TESTS 2024-03-09T12:55:43.888 INFO:tasks.workunit.client.0.smithi162.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2024-03-09T12:55:43.889 INFO:tasks.workunit.client.0.smithi162.stderr:++ uname 2024-03-09T12:55:43.890 INFO:tasks.workunit.client.0.smithi162.stderr:+ '[' Linux = FreeBSD ']' 2024-03-09T12:55:43.890 INFO:tasks.workunit.client.0.smithi162.stderr:+ use_ec_overwrite=true 2024-03-09T12:55:43.890 INFO:tasks.workunit.client.0.smithi162.stderr:+ getjson=no 2024-03-09T12:55:43.890 INFO:tasks.workunit.client.0.smithi162.stderr:+ jqfilter='def walk(f): 2024-03-09T12:55:43.890 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T12:55:43.890 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T12:55:43.890 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T12:55:43.890 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T12:55:43.890 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T12:55:43.890 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T12:55:43.890 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T12:55:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T12:55:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T12:55:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T12:55:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T12:55:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T12:55:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T12:55:43.896 INFO:tasks.workunit.client.0.smithi162.stderr:+ main osd-scrub-repair 2024-03-09T12:55:43.896 INFO:tasks.workunit.client.0.smithi162.stderr:+ local dir=td/osd-scrub-repair 2024-03-09T12:55:43.896 INFO:tasks.workunit.client.0.smithi162.stderr:+ shift 2024-03-09T12:55:43.896 INFO:tasks.workunit.client.0.smithi162.stderr:+ shopt -s -o xtrace 2024-03-09T12:55:43.896 INFO:tasks.workunit.client.0.smithi162.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-03-09T12:55:43.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/osd-scrub-repair 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2024-03-09T12:55:43.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2024-03-09T12:55:43.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none ' 2024-03-09T12:55:43.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2024-03-09T12:55:43.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2024-03-09T12:55:43.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2024-03-09T12:55:43.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2024-03-09T12:55:43.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-03-09T12:55:43.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_repair_during_recovery 2024-03-09T12:55:43.904 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_auto_repair_bluestore_basic 2024-03-09T12:55:43.904 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_auto_repair_bluestore_failed 2024-03-09T12:55:43.904 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_auto_repair_bluestore_failed_norecov 2024-03-09T12:55:43.904 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_auto_repair_bluestore_scrub 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_auto_repair_bluestore_tag 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_auto_repair_erasure_coded_appends 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_auto_repair_erasure_coded_overwrites 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_corrupt_and_repair_jerasure_appends 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_corrupt_and_repair_lrc_appends 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_corrupt_and_repair_lrc_overwrites 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_corrupt_and_repair_replicated 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_corrupt_scrub_erasure_appends 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_corrupt_scrub_erasure_overwrites 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_corrupt_scrub_replicated 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_corrupt_snapset_scrub_rep 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_list_missing_erasure_coded_appends 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_list_missing_erasure_coded_overwrites 2024-03-09T12:55:43.905 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_periodic_scrub_replicated 2024-03-09T12:55:43.906 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_repair_stats 2024-03-09T12:55:43.906 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_repair_stats_ec 2024-03-09T12:55:43.906 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_request_scrub_priority 2024-03-09T12:55:43.906 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_scrub_warning 2024-03-09T12:55:43.906 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_skip_non_repair_during_recovery 2024-03-09T12:55:43.906 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_unfound_erasure_coded_appends 2024-03-09T12:55:43.906 INFO:tasks.workunit.client.0.smithi162.stderr:TEST_unfound_erasure_coded_overwrites' 2024-03-09T12:55:43.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T12:55:43.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T12:55:43.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T12:55:43.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T12:55:43.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T12:55:43.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T12:55:43.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T12:55:43.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:55:43.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:55:43.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:55:43.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:55:43.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:55:43.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:55:43.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T12:55:43.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T12:55:43.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T12:55:43.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T12:55:43.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T12:55:43.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T12:55:43.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:55:43.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T12:55:43.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T12:55:43.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:55:43.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T12:55:43.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T12:55:43.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T12:55:43.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T12:55:43.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:55:43.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:55:43.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T12:55:43.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T12:55:43.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T12:55:43.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T12:55:43.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T12:55:43.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:55:43.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:55:43.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T12:55:43.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T12:55:43.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-03-09T12:55:43.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-03-09T12:55:43.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T12:55:43.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T12:55:43.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2024-03-09T12:55:43.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:114: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2024-03-09T12:55:43.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_repair_during_recovery: local poolname=rbd 2024-03-09T12:55:43.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:117: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-09T12:55:43.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T12:55:43.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T12:55:43.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T12:55:43.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T12:55:43.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T12:55:43.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-09T12:55:44.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T12:55:44.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:55:44.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:55:44.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:55:44.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:55:44.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:55:44.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:55:44.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-09T12:55:44.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T12:55:44.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T12:55:44.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T12:55:44.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T12:55:44.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T12:55:44.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T12:55:44.566 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T12:55:44.566 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T12:55:44.566 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T12:55:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T12:55:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:55:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:55:44.567 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T12:55:44.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T12:55:44.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T12:55:44.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T12:55:44.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T12:55:44.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T12:55:44.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T12:55:44.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T12:55:44.647 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T12:55:44.647 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T12:55:44.647 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T12:55:44.647 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T12:55:44.647 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:55:44.647 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:55:44.647 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T12:55:44.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T12:55:44.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T12:55:44.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:118: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2024-03-09T12:55:44.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T12:55:44.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T12:55:44.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T12:55:44.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T12:55:44.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T12:55:44.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T12:55:45.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T12:55:45.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:55:45.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:55:45.533 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:55:45.533 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:55:45.533 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:55:45.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:55:45.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T12:55:45.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T12:55:45.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:119: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-09T12:55:45.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T12:55:45.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:55:45.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:55:45.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:55:45.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:55:45.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-03-09T12:55:45.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T12:55:45.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:55:45.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e652f4b1-3e20-4307-b076-f84237e1f908 2024-03-09T12:55:45.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e652f4b1-3e20-4307-b076-f84237e1f908' 2024-03-09T12:55:45.566 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 e652f4b1-3e20-4307-b076-f84237e1f908 2024-03-09T12:55:45.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:55:45.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBRXOxlxTLEIhAAEwSyIBQmO6pjK1wB3p5Iag== 2024-03-09T12:55:45.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBRXOxlxTLEIhAAEwSyIBQmO6pjK1wB3p5Iag=="}' 2024-03-09T12:55:45.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e652f4b1-3e20-4307-b076-f84237e1f908 -i td/osd-scrub-repair/0/new.json 2024-03-09T12:55:45.788 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:55:45.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T12:55:45.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBRXOxlxTLEIhAAEwSyIBQmO6pjK1wB3p5Iag== --osd-uuid e652f4b1-3e20-4307-b076-f84237e1f908 2024-03-09T12:55:45.830 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:55:45.832+0000 7fb512a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:55:45.839 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:55:45.842+0000 7fb512a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:55:45.840 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:55:45.843+0000 7fb512a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:55:45.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:55:45.843+0000 7fb512a0f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T12:55:45.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:55:45.843+0000 7fb512a0f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T12:55:45.841 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:55:45.843+0000 7fb512a0f640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T12:55:45.923 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:55:45.925+0000 7fb512a0f640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T12:55:51.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T12:55:51.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:55:51.495 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T12:55:51.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T12:55:51.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:55:51.851 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T12:55:51.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T12:55:51.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-09T12:55:51.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:55:51.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:55:51.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:55:51.879 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:55:51.880+0000 7fa686e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:55:51.881 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:55:51.883+0000 7fa686e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:55:51.882 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:55:51.884+0000 7fa686e0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:55:52.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T12:55:52.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:55:52.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T12:55:52.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:55:52.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:55:52.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:55:52.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:55:52.127 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:55:52.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:55:52.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:55:52.399 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:55:52.401+0000 7fa686e0b640 -1 Falling back to public interface 2024-03-09T12:55:52.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:55:53.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:55:53.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:55:53.402 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:55:53.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:55:53.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:55:53.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:55:53.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:55:54.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:55:54.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:55:54.686 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:55:54.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:55:54.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:55:54.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:55:54.768 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:55:54.770+0000 7fa686e0b640 -1 osd.0 0 log_to_monitors true 2024-03-09T12:55:55.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:55:56.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:55:56.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:55:56.030 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:55:56.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:55:56.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:55:56.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:55:56.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:55:57.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:55:57.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:55:57.315 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T12:55:57.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T12:55:57.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:55:57.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:55:57.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:55:58.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:55:58.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:55:58.698 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T12:55:58.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T12:55:58.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:55:58.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:55:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:55:59.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:55:59.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:55:59.983 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T12:55:59.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T12:55:59.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:55:59.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:56:00.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:01.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:01.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:01.321 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T12:56:01.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T12:56:01.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:01.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:56:01.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:02.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:02.683 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T12:56:02.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:02.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T12:56:02.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:02.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:56:02.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:03.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:03.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:03.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T12:56:03.969 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T12:56:03.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:03.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:56:04.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:05.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:05.310 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T12:56:05.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:05.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T12:56:05.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:05.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:56:05.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:06.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:06.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:06.687 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T12:56:06.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T12:56:06.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:06.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:56:07.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:08.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:08.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:08.107 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T12:56:08.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T12:56:08.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:08.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:56:08.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:09.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:09.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:09.621 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T12:56:09.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T12:56:09.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:09.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:56:10.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:11.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:11.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:11.306 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T12:56:11.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T12:56:11.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:11.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:56:11.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:12.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:12.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:12.903 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T12:56:12.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T12:56:12.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:12.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:56:13.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:14.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:14.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:14.186 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T12:56:14.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T12:56:14.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:14.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:56:14.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:15.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:15.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:15.503 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T12:56:15.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T12:56:15.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:15.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:56:16.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T12:56:17.214 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T12:56:17.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:17.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:56:17.567 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/934906396,v1:127.0.0.1:6803/934906396] [v2:127.0.0.1:6804/934906396,v1:127.0.0.1:6805/934906396] exists,up e652f4b1-3e20-4307-b076-f84237e1f908 2024-03-09T12:56:17.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:56:17.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:56:17.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:56:17.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:122: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-09T12:56:17.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T12:56:17.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T12:56:17.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T12:56:17.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T12:56:17.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T12:56:17.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T12:56:17.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:56:17.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:56:17.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:56:17.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T12:56:17.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T12:56:17.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T12:56:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T12:56:17.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T12:56:17.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:56:17.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:56:17.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:56:17.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:56:17.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T12:56:17.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true' 2024-03-09T12:56:17.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T12:56:17.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T12:56:17.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a2c1482b-f5ae-4528-ac74-0de59c46330a 2024-03-09T12:56:17.574 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 a2c1482b-f5ae-4528-ac74-0de59c46330a 2024-03-09T12:56:17.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a2c1482b-f5ae-4528-ac74-0de59c46330a' 2024-03-09T12:56:17.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T12:56:17.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxXOxlPwk6IxAAXi46GLdVWMm5InmaEmvQLg== 2024-03-09T12:56:17.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxXOxlPwk6IxAAXi46GLdVWMm5InmaEmvQLg=="}' 2024-03-09T12:56:17.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a2c1482b-f5ae-4528-ac74-0de59c46330a -i td/osd-scrub-repair/1/new.json 2024-03-09T12:56:17.955 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:56:17.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T12:56:17.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true --mkfs --key AQBxXOxlPwk6IxAAXi46GLdVWMm5InmaEmvQLg== --osd-uuid a2c1482b-f5ae-4528-ac74-0de59c46330a 2024-03-09T12:56:17.998 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:56:18.000+0000 7fbaa3819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:56:18.040 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:56:18.042+0000 7fbaa3819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:56:18.040 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:56:18.043+0000 7fbaa3819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:56:18.041 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:56:18.044+0000 7fbaa3819640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T12:56:18.041 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:56:18.044+0000 7fbaa3819640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T12:56:18.041 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:56:18.044+0000 7fbaa3819640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T12:56:18.116 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:56:18.118+0000 7fbaa3819640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T12:56:23.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T12:56:23.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T12:56:23.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T12:56:23.744 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T12:56:23.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T12:56:24.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T12:56:24.093 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T12:56:24.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_repair_during_recovery=true --osd_debug_pretend_recovery_active=true 2024-03-09T12:56:24.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T12:56:24.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T12:56:24.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T12:56:24.121 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:56:24.124+0000 7f2d68521640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:56:24.123 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:56:24.126+0000 7f2d68521640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:56:24.124 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:56:24.127+0000 7f2d68521640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:56:24.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T12:56:24.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:56:24.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T12:56:24.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:56:24.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:56:24.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:24.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:56:24.368 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:56:24.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:24.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:24.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:24.643 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:56:24.646+0000 7f2d68521640 -1 Falling back to public interface 2024-03-09T12:56:25.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:25.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:25.643 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:56:25.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:56:25.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:25.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:25.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:26.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:26.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:26.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:56:26.923 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:56:26.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:26.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:26.968 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:56:26.970+0000 7f2d68521640 -1 osd.1 0 log_to_monitors true 2024-03-09T12:56:27.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:28.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:28.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:28.269 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:56:28.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:56:28.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:28.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:28.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:29.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:29.551 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T12:56:29.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:29.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T12:56:29.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:29.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:30.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:31.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:31.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:31.041 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T12:56:31.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T12:56:31.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:31.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:31.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:32.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:32.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:32.322 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T12:56:32.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T12:56:32.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:32.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:32.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:33.611 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T12:56:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T12:56:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:33.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:35.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:35.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:35.000 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T12:56:35.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T12:56:35.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:35.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:35.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:36.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:36.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:36.288 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T12:56:36.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T12:56:36.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:36.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:36.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:37.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:37.594 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T12:56:37.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:37.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T12:56:37.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:37.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:38.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:39.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:39.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:39.044 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T12:56:39.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T12:56:39.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:39.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:39.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:40.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:40.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:40.326 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T12:56:40.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T12:56:40.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:40.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:40.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:41.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:41.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:41.645 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T12:56:41.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T12:56:41.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:41.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:42.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:43.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:43.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:43.389 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T12:56:43.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T12:56:43.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:43.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:43.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:44.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:44.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:44.836 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T12:56:44.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T12:56:44.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:44.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:45.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:46.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:46.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:46.269 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T12:56:46.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T12:56:46.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:46.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:46.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:47.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:47.711 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T12:56:47.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:47.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T12:56:47.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:47.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:48.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:49.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:49.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:49.442 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T12:56:49.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T12:56:49.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:49.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:49.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:56:50.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:56:50.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:56:50.816 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T12:56:50.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T12:56:50.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:56:50.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:56:51.120 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1368816044,v1:127.0.0.1:6811/1368816044] [v2:127.0.0.1:6812/1368816044,v1:127.0.0.1:6813/1368816044] exists,up a2c1482b-f5ae-4528-ac74-0de59c46330a 2024-03-09T12:56:51.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:56:51.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:56:51.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:56:51.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_repair_during_recovery: create_rbd_pool 2024-03-09T12:56:51.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T12:56:51.385 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T12:56:51.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T12:56:51.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T12:56:51.801 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T12:56:51.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T12:56:52.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T12:56:54.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_repair_during_recovery: wait_for_clean 2024-03-09T12:56:54.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T12:56:54.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T12:56:54.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T12:56:54.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T12:56:54.686 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T12:56:54.686 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T12:56:54.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T12:56:54.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T12:56:54.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T12:56:54.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T12:56:54.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T12:56:54.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T12:56:54.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T12:56:54.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T12:56:54.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T12:56:55.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T12:56:55.069 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T12:56:55.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T12:56:55.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:56:55.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T12:56:55.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-03-09T12:56:55.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-03-09T12:56:55.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-03-09T12:56:55.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:56:55.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T12:56:55.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705672 2024-03-09T12:56:55.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705672 2024-03-09T12:56:55.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-38654705672' 2024-03-09T12:56:55.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:56:55.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-03-09T12:56:55.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:56:55.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T12:56:55.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-03-09T12:56:55.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:56:55.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-03-09T12:56:55.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-03-09T12:56:55.280 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836493 2024-03-09T12:56:55.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:56:55.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-03-09T12:56:55.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:56:56.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T12:56:56.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:56:56.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836493 2024-03-09T12:56:56.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:56:56.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705672 2024-03-09T12:56:56.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:56:56.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T12:56:56.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705672 2024-03-09T12:56:56.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:56:56.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705672 2024-03-09T12:56:56.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2024-03-09T12:56:56.838 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 38654705672 2024-03-09T12:56:56.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T12:56:57.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705673 -lt 38654705672 2024-03-09T12:56:57.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T12:56:57.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:56:57.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:56:57.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T12:56:57.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T12:56:57.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T12:56:57.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T12:56:57.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T12:56:57.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T12:56:57.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T12:56:57.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T12:56:57.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T12:56:57.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T12:56:57.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:56:57.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:56:58.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T12:56:58.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T12:56:58.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T12:56:58.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2024-03-09T12:56:58.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T12:56:58.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-03-09T12:56:58.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-09T12:56:58.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T12:56:58.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T12:56:58.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T12:56:58.426 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T12:56:58.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T12:56:58.954 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T12:56:58.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T12:56:58.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T12:56:58.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T12:56:59.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2024-03-09T12:56:59.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-03-09T12:56:59.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T12:56:59.151 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-03-09T12:56:59.151 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-09T12:56:59.151 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T12:56:59.151 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-09T12:56:59.151 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T12:56:59.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T12:56:59.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-03-09T12:56:59.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T12:56:59.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:129: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-03-09T12:56:59.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T12:56:59.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-09T12:56:59.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-03-09T12:56:59.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:56:59.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:56:59.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:57:00.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:57:00.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T12:57:00.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T12:57:00.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T12:57:00.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T12:57:00.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T12:57:00.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T12:57:00.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-09T12:57:01.600 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-09T12:57:02.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T12:57:02.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T12:57:02.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T12:57:02.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T12:57:02.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T12:57:02.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T12:57:02.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T12:57:02.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:57:02.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:57:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:57:02.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:57:02.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T12:57:02.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T12:57:02.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:57:02.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:57:02.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:57:02.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:57:02.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T12:57:02.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T12:57:02.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T12:57:02.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T12:57:02.120 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T12:57:02.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T12:57:02.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T12:57:02.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T12:57:02.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T12:57:02.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T12:57:02.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T12:57:02.147 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:02.150+0000 7f462280d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:57:02.149 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:02.152+0000 7f462280d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:57:02.150 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:02.153+0000 7f462280d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:57:02.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T12:57:02.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:57:02.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T12:57:02.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:57:02.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:57:02.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:57:02.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:57:02.400 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T12:57:02.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:57:02.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:57:02.665 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:02.668+0000 7f462280d640 -1 Falling back to public interface 2024-03-09T12:57:02.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:57:03.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:57:03.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:57:03.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:57:03.677 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T12:57:03.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:57:03.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:57:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:57:04.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:57:04.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:57:04.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:57:04.958 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T12:57:04.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:57:04.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:57:05.057 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:05.060+0000 7f462280d640 -1 osd.0 19 log_to_monitors true 2024-03-09T12:57:05.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:57:06.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:57:06.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:57:06.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:57:06.269 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T12:57:06.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:57:06.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:57:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 24 up_thru 24 down_at 20 last_clean_interval [5,19) [v2:127.0.0.1:6802/4235636422,v1:127.0.0.1:6803/4235636422] [v2:127.0.0.1:6804/4235636422,v1:127.0.0.1:6805/4235636422] exists,up e652f4b1-3e20-4307-b076-f84237e1f908 2024-03-09T12:57:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:57:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:57:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:57:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T12:57:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T12:57:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T12:57:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T12:57:06.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T12:57:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T12:57:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T12:57:06.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T12:57:06.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T12:57:06.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T12:57:06.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T12:57:06.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T12:57:06.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T12:57:06.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T12:57:06.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T12:57:06.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T12:57:06.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T12:57:06.970 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T12:57:06.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T12:57:06.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:57:06.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T12:57:07.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-03-09T12:57:07.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-03-09T12:57:07.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-03-09T12:57:07.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:57:07.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T12:57:07.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=38654705676 2024-03-09T12:57:07.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 38654705676 2024-03-09T12:57:07.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-38654705676' 2024-03-09T12:57:07.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:57:07.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-03-09T12:57:07.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:57:07.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T12:57:07.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-03-09T12:57:07.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:57:07.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-03-09T12:57:07.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-03-09T12:57:07.180 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 103079215106 2024-03-09T12:57:07.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:57:07.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-03-09T12:57:07.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:57:08.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T12:57:08.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:57:08.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-03-09T12:57:08.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:57:08.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-38654705676 2024-03-09T12:57:08.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:57:08.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T12:57:08.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-38654705676 2024-03-09T12:57:08.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:57:08.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=38654705676 2024-03-09T12:57:08.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2024-03-09T12:57:08.738 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 38654705676 2024-03-09T12:57:08.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T12:57:09.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 38654705676 -lt 38654705676 2024-03-09T12:57:09.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T12:57:09.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:57:09.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:57:09.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T12:57:09.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T12:57:09.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T12:57:09.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T12:57:09.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T12:57:09.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T12:57:09.357 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T12:57:09.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T12:57:09.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T12:57:09.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T12:57:09.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:57:09.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:57:09.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T12:57:09.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T12:57:09.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T12:57:09.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-09T12:57:09.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-09T12:57:09.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T12:57:09.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-09T12:57:09.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T12:57:10.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-09T12:57:10.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-09T12:57:10.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-09T12:57:10.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-09T12:57:10.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T12:57:10.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:57:10.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:57:10.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T12:57:10.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T12:56:51.753515+0000 2024-03-09T12:57:10.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-09T12:57:10.787 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-09T12:57:10.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-09T12:56:51.753515+0000 2024-03-09T12:57:10.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-09T12:57:10.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T12:56:51.753515+0000 2024-03-09T12:57:10.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T12:57:10.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T12:57:10.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T12:57:10.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T12:57:10.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T12:57:10.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:57:10.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:57:10.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T12:57:11.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T12:56:51.753515+0000 '>' 2024-03-09T12:56:51.753515+0000 2024-03-09T12:57:11.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T12:57:12.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T12:57:12.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T12:57:12.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T12:57:12.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T12:57:12.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:57:12.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:57:12.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T12:57:12.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T12:56:51.753515+0000 '>' 2024-03-09T12:56:51.753515+0000 2024-03-09T12:57:12.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T12:57:13.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T12:57:13.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T12:57:13.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T12:57:13.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T12:57:13.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:57:13.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:57:13.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T12:57:13.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T12:56:51.753515+0000 '>' 2024-03-09T12:56:51.753515+0000 2024-03-09T12:57:13.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T12:57:14.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T12:57:14.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T12:57:14.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T12:57:14.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T12:57:14.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:57:14.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:57:14.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T12:57:14.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T12:56:51.753515+0000 '>' 2024-03-09T12:56:51.753515+0000 2024-03-09T12:57:14.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T12:57:15.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T12:57:15.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T12:57:15.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T12:57:15.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T12:57:15.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:57:15.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:57:15.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T12:57:16.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T12:57:11.785818+0000 '>' 2024-03-09T12:56:51.753515+0000 2024-03-09T12:57:16.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T12:57:16.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T12:57:16.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T12:57:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T12:57:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T12:57:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T12:57:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T12:57:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T12:57:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T12:57:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T12:57:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T12:57:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T12:57:16.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:57:16.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:57:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:57:16.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:57:16.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:57:16.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:57:16.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T12:57:16.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T12:57:16.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T12:57:16.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T12:57:16.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T12:57:16.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T12:57:16.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T12:57:17.581 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-09T12:57:17.581 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T12:57:17.581 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T12:57:17.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T12:57:17.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T12:57:17.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T12:57:17.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:57:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T12:57:17.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T12:57:17.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T12:57:17.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T12:57:17.852 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T12:57:17.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T12:57:17.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T12:57:17.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T12:57:17.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T12:57:17.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T12:57:17.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T12:57:17.878 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:17.881+0000 7f1f90791640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:57:17.880 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:17.883+0000 7f1f90791640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:57:17.882 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:17.885+0000 7f1f90791640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:57:18.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T12:57:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:57:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T12:57:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:57:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:57:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:57:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:57:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T12:57:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:57:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:57:18.398 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:18.401+0000 7f1f90791640 -1 Falling back to public interface 2024-03-09T12:57:18.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:57:19.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:57:19.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:57:19.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:57:19.401 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T12:57:19.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:57:19.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:57:19.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:57:20.593 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:20.596+0000 7f1f90791640 -1 osd.1 26 log_to_monitors true 2024-03-09T12:57:20.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:57:20.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:57:20.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:57:20.732 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T12:57:20.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:57:20.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T12:57:21.012 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 30 up_thru 25 down_at 27 last_clean_interval [9,26) [v2:127.0.0.1:6810/1477403836,v1:127.0.0.1:6811/1477403836] [v2:127.0.0.1:6812/1477403836,v1:127.0.0.1:6813/1477403836] exists,up a2c1482b-f5ae-4528-ac74-0de59c46330a 2024-03-09T12:57:21.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:57:21.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:57:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:57:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T12:57:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T12:57:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T12:57:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T12:57:21.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T12:57:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T12:57:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T12:57:21.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T12:57:21.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T12:57:21.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T12:57:21.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T12:57:21.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T12:57:21.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T12:57:21.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T12:57:21.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T12:57:21.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T12:57:21.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T12:57:21.490 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T12:57:21.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T12:57:21.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:57:21.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T12:57:21.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215110 2024-03-09T12:57:21.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215110 2024-03-09T12:57:21.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110' 2024-03-09T12:57:21.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:57:21.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T12:57:21.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-09T12:57:21.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-09T12:57:21.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110 1-128849018882' 2024-03-09T12:57:21.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:57:21.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215110 2024-03-09T12:57:21.746 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:57:21.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T12:57:21.746 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215110 2024-03-09T12:57:21.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:57:21.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215110 2024-03-09T12:57:21.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2024-03-09T12:57:21.747 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 103079215110 2024-03-09T12:57:21.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:57:22.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215110 2024-03-09T12:57:22.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:57:23.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T12:57:23.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:57:23.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215110 2024-03-09T12:57:23.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:57:24.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T12:57:24.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:57:24.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215110 -lt 103079215110 2024-03-09T12:57:24.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:57:24.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-128849018882 2024-03-09T12:57:24.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:57:24.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T12:57:24.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-128849018882 2024-03-09T12:57:24.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:57:24.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-09T12:57:24.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2024-03-09T12:57:24.593 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 128849018882 2024-03-09T12:57:24.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T12:57:24.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-09T12:57:24.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T12:57:24.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:57:24.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:57:25.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T12:57:25.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T12:57:25.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T12:57:25.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T12:57:25.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T12:57:25.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T12:57:25.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T12:57:25.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T12:57:25.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T12:57:25.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T12:57:25.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:57:25.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:57:25.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T12:57:25.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T12:57:25.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T12:57:25.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T12:57:25.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T12:57:25.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T12:57:25.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T12:57:25.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T12:57:25.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T12:57:25.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:57:25.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:57:25.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:57:25.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:57:25.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:57:25.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:57:25.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T12:57:25.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T12:57:25.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T12:57:25.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T12:57:25.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T12:57:25.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T12:57:25.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:57:25.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T12:57:25.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T12:57:25.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:57:25.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T12:57:25.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T12:57:25.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T12:57:25.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T12:57:25.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:57:25.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:57:25.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T12:57:25.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T12:57:25.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T12:57:25.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T12:57:25.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T12:57:25.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T12:57:25.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T12:57:25.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T12:57:25.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T12:57:25.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T12:57:25.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:57:25.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:57:25.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:57:25.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:57:25.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:57:25.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:57:25.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T12:57:25.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T12:57:25.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T12:57:25.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T12:57:25.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T12:57:25.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T12:57:25.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:57:25.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T12:57:25.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T12:57:25.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T12:57:25.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T12:57:25.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T12:57:25.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T12:57:25.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T12:57:25.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:57:25.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:57:25.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T12:57:25.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T12:57:25.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T12:57:25.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T12:57:25.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T12:57:25.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:57:25.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:57:25.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T12:57:25.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T12:57:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T12:57:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T12:57:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T12:57:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2024-03-09T12:57:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2024-03-09T12:57:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2024-03-09T12:57:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:439: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2024-03-09T12:57:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:445: TEST_auto_repair_bluestore_basic: local extr_dbg=3 2024-03-09T12:57:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:446: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2024-03-09T12:57:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:221: standard_scrub_cluster: local dir=td/osd-scrub-repair 2024-03-09T12:57:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:222: standard_scrub_cluster: local -n args=cluster_conf 2024-03-09T12:57:26.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:224: standard_scrub_cluster: local OSDS=3 2024-03-09T12:57:26.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:225: standard_scrub_cluster: local pg_num=1 2024-03-09T12:57:26.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:226: standard_scrub_cluster: local poolname=testpool 2024-03-09T12:57:26.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:227: standard_scrub_cluster: args['pool_name']=testpool 2024-03-09T12:57:26.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:228: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2024-03-09T12:57:26.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local debug_msg=dbg 2024-03-09T12:57:26.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local saved_echo_flag=x 2024-03-09T12:57:26.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: set +x 2024-03-09T12:57:27.626 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 5c72022f-cc7f-4f92-b554-436d9465a9d6 2024-03-09T12:57:27.848 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:57:27.891 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:27.893+0000 7f2a5aa15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:57:27.899 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:27.901+0000 7f2a5aa15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:57:27.900 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:27.902+0000 7f2a5aa15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:57:27.900 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:27.902+0000 7f2a5aa15640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T12:57:27.900 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:27.902+0000 7f2a5aa15640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T12:57:27.900 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:27.902+0000 7f2a5aa15640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T12:57:28.016 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:28.018+0000 7f2a5aa15640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T12:57:33.534 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T12:57:33.899 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T12:57:33.927 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:33.929+0000 7f913ee01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:57:33.928 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:33.931+0000 7f913ee01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:57:33.930 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:33.932+0000 7f913ee01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:57:34.171 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:57:34.461 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:34.463+0000 7f913ee01640 -1 Falling back to public interface 2024-03-09T12:57:35.442 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:57:36.718 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:57:36.789 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:57:36.790+0000 7f913ee01640 -1 osd.0 0 log_to_monitors true 2024-03-09T12:57:38.040 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:57:39.327 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T12:57:40.739 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T12:57:42.021 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T12:57:43.346 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T12:57:44.732 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T12:57:46.010 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T12:57:47.325 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T12:57:48.613 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T12:57:50.262 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T12:57:51.913 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T12:57:53.565 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T12:57:55.367 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T12:57:56.826 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T12:57:58.249 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T12:57:59.832 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T12:58:00.128 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/375817491,v1:127.0.0.1:6803/375817491] [v2:127.0.0.1:6804/375817491,v1:127.0.0.1:6805/375817491] exists,up 5c72022f-cc7f-4f92-b554-436d9465a9d6 2024-03-09T12:58:00.132 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 98cbfc27-14b6-4d01-9f99-b58f19f61e2a 2024-03-09T12:58:00.516 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:58:00.560 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:00.563+0000 7f8cbcc0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:58:00.601 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:00.604+0000 7f8cbcc0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:58:00.602 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:00.605+0000 7f8cbcc0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:58:00.603 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:00.605+0000 7f8cbcc0f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T12:58:00.603 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:00.605+0000 7f8cbcc0f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T12:58:00.603 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:00.605+0000 7f8cbcc0f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T12:58:00.676 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:00.679+0000 7f8cbcc0f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T12:58:06.127 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T12:58:06.479 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T12:58:06.511 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:06.514+0000 7f420418c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:58:06.513 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:06.516+0000 7f420418c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:58:06.514 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:06.517+0000 7f420418c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:58:06.755 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:58:07.047 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:07.050+0000 7f420418c640 -1 Falling back to public interface 2024-03-09T12:58:08.033 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:58:09.316 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:58:09.381 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:09.384+0000 7f420418c640 -1 osd.1 0 log_to_monitors true 2024-03-09T12:58:10.592 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:58:11.880 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T12:58:13.166 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T12:58:14.453 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T12:58:15.737 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T12:58:17.076 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T12:58:18.399 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T12:58:19.730 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T12:58:21.076 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T12:58:22.836 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T12:58:24.507 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T12:58:25.891 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T12:58:27.576 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T12:58:29.145 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T12:58:30.768 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T12:58:32.332 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T12:58:32.614 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/731847151,v1:127.0.0.1:6811/731847151] [v2:127.0.0.1:6812/731847151,v1:127.0.0.1:6813/731847151] exists,up 98cbfc27-14b6-4d01-9f99-b58f19f61e2a 2024-03-09T12:58:32.617 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 b0f09983-54d4-4e02-bf24-0b167102dbf5 2024-03-09T12:58:32.953 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:58:32.998 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:32.999+0000 7f8343c68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:58:33.038 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:33.040+0000 7f8343c68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:58:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:33.041+0000 7f8343c68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:58:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:33.041+0000 7f8343c68640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T12:58:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:33.041+0000 7f8343c68640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T12:58:33.039 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:33.041+0000 7f8343c68640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T12:58:33.113 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:33.115+0000 7f8343c68640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T12:58:39.141 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T12:58:39.496 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T12:58:39.527 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:39.530+0000 7fb7e6a1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:58:39.529 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:39.532+0000 7fb7e6a1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:58:39.530 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:39.533+0000 7fb7e6a1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:58:39.822 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T12:58:40.056 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:40.059+0000 7fb7e6a1e640 -1 Falling back to public interface 2024-03-09T12:58:41.100 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T12:58:42.382 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T12:58:42.420 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:58:42.423+0000 7fb7e6a1e640 -1 osd.2 0 log_to_monitors true 2024-03-09T12:58:43.666 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T12:58:44.946 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T12:58:46.275 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T12:58:47.611 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T12:58:48.948 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T12:58:50.234 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T12:58:51.603 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T12:58:52.966 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T12:58:54.253 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T12:58:55.541 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T12:58:57.227 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T12:58:58.858 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T12:59:00.442 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T12:59:02.053 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T12:59:03.361 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T12:59:04.789 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T12:59:06.269 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T12:59:06.546 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1949802654,v1:127.0.0.1:6819/1949802654] [v2:127.0.0.1:6820/1949802654,v1:127.0.0.1:6821/1949802654] exists,up b0f09983-54d4-4e02-bf24-0b167102dbf5 2024-03-09T12:59:06.919 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'testpool' created 2024-03-09T12:59:08.803 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836498 2024-03-09T12:59:10.405 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672975 2024-03-09T12:59:10.681 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509447 2024-03-09T12:59:16.456 INFO:tasks.workunit.client.0.smithi162.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2024-03-09T12:59:16.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_basic: local poolid=1 2024-03-09T12:59:16.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_basic: local poolname=testpool 2024-03-09T12:59:16.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:450: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2024-03-09T12:59:16.824 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 1 size to 2 2024-03-09T12:59:16.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:451: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-03-09T12:59:16.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T12:59:16.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T12:59:16.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T12:59:16.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T12:59:16.837 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T12:59:16.837 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T12:59:16.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T12:59:16.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T12:59:16.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T12:59:16.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T12:59:16.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T12:59:16.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T12:59:16.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T12:59:16.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T12:59:16.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T12:59:17.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T12:59:17.411 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T12:59:17.411 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T12:59:17.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T12:59:17.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:59:17.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T12:59:17.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-03-09T12:59:17.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-03-09T12:59:17.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-03-09T12:59:17.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:59:17.515 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T12:59:17.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-03-09T12:59:17.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-03-09T12:59:17.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672977' 2024-03-09T12:59:17.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:59:17.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T12:59:17.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-03-09T12:59:17.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-03-09T12:59:17.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672977 2-64424509450' 2024-03-09T12:59:17.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:59:17.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-03-09T12:59:17.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:59:17.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T12:59:17.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-03-09T12:59:17.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:59:17.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-03-09T12:59:17.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-03-09T12:59:17.782 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836501 2024-03-09T12:59:17.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:59:18.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-03-09T12:59:18.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:59:18.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-03-09T12:59:18.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:59:18.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T12:59:18.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-03-09T12:59:18.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:59:18.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-03-09T12:59:18.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-03-09T12:59:18.058 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672977 2024-03-09T12:59:18.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T12:59:18.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672977 2024-03-09T12:59:18.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:59:19.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T12:59:19.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T12:59:19.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672977 2024-03-09T12:59:19.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:59:20.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T12:59:20.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T12:59:20.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672977 2024-03-09T12:59:20.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:59:20.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-03-09T12:59:20.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:59:20.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T12:59:20.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-03-09T12:59:20.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:59:20.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-03-09T12:59:20.898 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509450 2024-03-09T12:59:20.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-03-09T12:59:20.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T12:59:21.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-03-09T12:59:21.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T12:59:21.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:59:21.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:59:21.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T12:59:21.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T12:59:21.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T12:59:21.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T12:59:21.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T12:59:21.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T12:59:21.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T12:59:21.518 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T12:59:21.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T12:59:21.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T12:59:21.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:59:21.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:59:22.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T12:59:22.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T12:59:22.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T12:59:22.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:454: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2024-03-09T12:59:22.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_basic: echo ABCDEF 2024-03-09T12:59:22.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T12:59:22.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-09T12:59:22.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T12:59:22.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T12:59:22.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T12:59:22.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T12:59:22.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T12:59:22.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T12:59:22.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T12:59:22.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T12:59:22.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T12:59:22.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T12:59:22.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T12:59:22.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T12:59:22.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T12:59:22.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T12:59:22.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T12:59:22.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T12:59:22.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T12:59:22.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T12:59:22.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T12:59:22.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T12:59:22.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T12:59:22.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:59:22.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:59:22.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:59:22.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:59:22.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:59:23.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:59:23.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T12:59:23.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T12:59:23.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T12:59:23.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T12:59:23.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T12:59:23.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T12:59:23.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-09T12:59:24.490 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-09T12:59:25.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T12:59:25.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T12:59:25.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T12:59:25.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T12:59:25.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T12:59:25.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T12:59:25.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T12:59:25.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:59:25.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:59:25.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:59:25.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T12:59:25.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T12:59:25.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T12:59:25.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T12:59:25.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T12:59:25.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T12:59:25.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:59:25.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:59:25.010 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:59:25.010 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:59:25.010 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T12:59:25.011 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T12:59:25.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T12:59:25.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T12:59:25.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T12:59:25.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T12:59:25.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T12:59:25.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T12:59:25.038 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:25.041+0000 7f38dd106640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:59:25.041 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:25.044+0000 7f38dd106640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:59:25.042 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:25.045+0000 7f38dd106640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:59:25.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T12:59:25.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:59:25.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T12:59:25.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:59:25.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:59:25.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:59:25.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:59:25.293 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T12:59:25.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:59:25.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:59:25.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:59:25.575 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:25.578+0000 7f38dd106640 -1 Falling back to public interface 2024-03-09T12:59:26.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:59:26.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:59:26.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:59:26.569 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T12:59:26.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:59:26.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:59:26.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:59:27.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:59:27.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:59:27.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:59:27.851 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T12:59:27.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:59:27.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:59:28.088 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:28.091+0000 7f38dd106640 -1 osd.0 21 log_to_monitors true 2024-03-09T12:59:28.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:59:29.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:59:29.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:59:29.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T12:59:29.184 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T12:59:29.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:59:29.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:59:29.466 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/1483977755,v1:127.0.0.1:6803/1483977755] [v2:127.0.0.1:6804/1483977755,v1:127.0.0.1:6805/1483977755] exists,up 5c72022f-cc7f-4f92-b554-436d9465a9d6 2024-03-09T12:59:29.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:59:29.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:59:29.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:59:29.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T12:59:29.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T12:59:29.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T12:59:29.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T12:59:29.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T12:59:29.466 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T12:59:29.467 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T12:59:29.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T12:59:29.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T12:59:29.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T12:59:29.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T12:59:29.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T12:59:29.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T12:59:29.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T12:59:29.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T12:59:29.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T12:59:29.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T12:59:29.835 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T12:59:29.835 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T12:59:29.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T12:59:29.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:59:29.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T12:59:29.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-09T12:59:29.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-09T12:59:29.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-09T12:59:29.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:59:29.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T12:59:30.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-03-09T12:59:30.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-03-09T12:59:30.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672981' 2024-03-09T12:59:30.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:59:30.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T12:59:30.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-09T12:59:30.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-09T12:59:30.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672981 2-64424509453' 2024-03-09T12:59:30.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:59:30.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-09T12:59:30.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:59:30.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T12:59:30.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-09T12:59:30.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:59:30.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-09T12:59:30.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-09T12:59:30.152 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 107374182402 2024-03-09T12:59:30.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:59:30.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-09T12:59:30.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:59:31.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T12:59:31.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:59:31.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-09T12:59:31.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:59:32.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T12:59:32.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:59:33.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-09T12:59:33.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:59:33.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-03-09T12:59:33.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:59:33.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T12:59:33.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-03-09T12:59:33.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:59:33.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-03-09T12:59:33.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-03-09T12:59:33.013 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672981 2024-03-09T12:59:33.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T12:59:33.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-03-09T12:59:33.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:59:33.285 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-09T12:59:33.285 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:59:33.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T12:59:33.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-09T12:59:33.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:59:33.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-09T12:59:33.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-09T12:59:33.288 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509453 2024-03-09T12:59:33.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T12:59:33.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2024-03-09T12:59:33.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T12:59:33.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:59:33.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:59:33.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T12:59:33.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T12:59:33.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T12:59:33.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T12:59:33.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T12:59:33.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T12:59:33.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T12:59:33.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T12:59:34.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T12:59:34.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T12:59:34.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:59:34.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:59:34.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T12:59:34.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T12:59:34.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T12:59:34.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-03-09T12:59:34.615 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0: { 2024-03-09T12:59:34.615 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-09T12:59:34.615 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:59:34.629 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1: { 2024-03-09T12:59:34.629 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-09T12:59:34.629 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:59:34.643 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2: { 2024-03-09T12:59:34.643 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-09T12:59:34.644 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:59:34.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2024-03-09T12:59:34.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T12:59:34.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T12:59:34.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-09T12:59:34.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T12:59:34.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:463: TEST_auto_repair_bluestore_basic: local pgid=1.0 2024-03-09T12:59:34.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2024-03-09T12:59:34.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T12:59:34.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T12:59:34.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T12:59:34.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T12:59:35.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:464: TEST_auto_repair_bluestore_basic: local primary=1 2024-03-09T12:59:35.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2024-03-09T12:59:35.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T12:59:35.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:59:35.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:59:35.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T12:59:35.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2024-03-09T12:59:06.866906+0000 2024-03-09T12:59:35.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:468: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2024-03-09T12:59:35.576 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T12:59:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": true, 2024-03-09T12:59:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: "must": false, 2024-03-09T12:59:35.576 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "2024-02-24T12:57:55.579786+0000" 2024-03-09T12:59:35.576 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T12:59:35.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2024-03-09T12:59:06.866906+0000 2024-03-09T12:59:35.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T12:59:35.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T12:59:06.866906+0000 2024-03-09T12:59:35.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T12:59:35.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T12:59:35.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T12:59:35.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T12:59:35.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T12:59:35.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:59:35.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:59:35.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T12:59:35.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T12:59:06.866906+0000 '>' 2024-03-09T12:59:06.866906+0000 2024-03-09T12:59:35.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T12:59:36.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T12:59:36.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T12:59:36.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T12:59:36.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T12:59:36.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:59:36.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:59:36.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T12:59:37.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T12:59:06.866906+0000 '>' 2024-03-09T12:59:06.866906+0000 2024-03-09T12:59:37.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T12:59:38.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T12:59:38.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T12:59:38.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T12:59:38.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T12:59:38.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:59:38.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:59:38.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T12:59:38.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T12:59:06.866906+0000 '>' 2024-03-09T12:59:06.866906+0000 2024-03-09T12:59:38.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T12:59:39.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T12:59:39.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T12:59:39.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T12:59:39.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T12:59:39.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T12:59:39.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T12:59:39.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T12:59:39.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T12:59:36.504635+0000 '>' 2024-03-09T12:59:06.866906+0000 2024-03-09T12:59:39.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T12:59:39.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:472: TEST_auto_repair_bluestore_basic: wait_for_clean 2024-03-09T12:59:39.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T12:59:39.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T12:59:39.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T12:59:39.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T12:59:39.701 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T12:59:39.701 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T12:59:39.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T12:59:39.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T12:59:39.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T12:59:39.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T12:59:39.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T12:59:39.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T12:59:39.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T12:59:39.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T12:59:39.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T12:59:40.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T12:59:40.059 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T12:59:40.059 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T12:59:40.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T12:59:40.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:59:40.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T12:59:40.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182405 2024-03-09T12:59:40.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182405 2024-03-09T12:59:40.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405' 2024-03-09T12:59:40.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:59:40.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T12:59:40.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-03-09T12:59:40.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-03-09T12:59:40.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672984' 2024-03-09T12:59:40.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:59:40.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T12:59:40.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-09T12:59:40.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-09T12:59:40.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182405 1-42949672984 2-64424509456' 2024-03-09T12:59:40.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:59:40.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182405 2024-03-09T12:59:40.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:59:40.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T12:59:40.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182405 2024-03-09T12:59:40.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:59:40.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182405 2024-03-09T12:59:40.373 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 107374182405 2024-03-09T12:59:40.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182405' 2024-03-09T12:59:40.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:59:40.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182404 -lt 107374182405 2024-03-09T12:59:40.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:59:41.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T12:59:41.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:59:41.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182405 -lt 107374182405 2024-03-09T12:59:41.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:59:41.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-03-09T12:59:41.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:59:41.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T12:59:41.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-03-09T12:59:41.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:59:41.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-03-09T12:59:41.935 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672984 2024-03-09T12:59:41.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-03-09T12:59:41.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T12:59:42.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-03-09T12:59:42.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:59:42.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-09T12:59:42.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:59:42.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T12:59:42.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-09T12:59:42.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:59:42.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-09T12:59:42.211 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509456 2024-03-09T12:59:42.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-09T12:59:42.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T12:59:42.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509456 2024-03-09T12:59:42.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T12:59:42.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:59:42.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:59:42.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T12:59:42.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T12:59:42.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T12:59:42.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T12:59:42.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T12:59:42.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T12:59:42.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T12:59:42.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T12:59:43.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T12:59:43.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T12:59:43.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:59:43.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:59:43.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T12:59:43.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T12:59:43.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T12:59:43.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2024-03-09T12:59:43.688 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T12:59:43.689 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-09T12:59:36.694535+0000 21'1 26:57 [1,0] 1 [1,0] 1 21'1 2024-03-09T12:59:36.504635+0000 21'1 2024-03-09T12:59:36.504635+0000 0 1 periodic scrub scheduled @ 2024-03-10T12:59:36.504635+0000 1 0 2024-03-09T12:59:43.689 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T12:59:43.689 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T12:59:43.689 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T12:59:43.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-09T12:59:43.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T12:59:43.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T12:59:43.706 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T12:59:43.706 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T12:59:43.706 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T12:59:43.706 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T12:59:43.706 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T12:59:43.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T12:59:43.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T12:59:43.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T12:59:44.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T12:59:44.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T12:59:44.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T12:59:44.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T12:59:44.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T12:59:44.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T12:59:44.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T12:59:44.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T12:59:44.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T12:59:44.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T12:59:44.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T12:59:44.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:59:44.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:59:44.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:59:44.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:59:44.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:59:44.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:59:44.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T12:59:44.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T12:59:44.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T12:59:44.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T12:59:44.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T12:59:44.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T12:59:44.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-09T12:59:45.803 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T12:59:45.803 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T12:59:46.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T12:59:46.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T12:59:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T12:59:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T12:59:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T12:59:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T12:59:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T12:59:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:59:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:59:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:59:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T12:59:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T12:59:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T12:59:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T12:59:46.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:59:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:59:46.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T12:59:46.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T12:59:46.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T12:59:46.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T12:59:46.073 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T12:59:46.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T12:59:46.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T12:59:46.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T12:59:46.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T12:59:46.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T12:59:46.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T12:59:46.100 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:46.102+0000 7f7edd711640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:59:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:46.104+0000 7f7edd711640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:59:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:46.105+0000 7f7edd711640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:59:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T12:59:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:59:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T12:59:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:59:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:59:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:59:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:59:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T12:59:46.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:59:46.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:59:46.618 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:46.620+0000 7f7edd711640 -1 Falling back to public interface 2024-03-09T12:59:46.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:59:47.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:59:47.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:59:47.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:59:47.627 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T12:59:47.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:59:47.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:59:47.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:59:48.849 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:48.850+0000 7f7edd711640 -1 osd.0 26 log_to_monitors true 2024-03-09T12:59:48.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:59:48.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:59:48.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:59:48.919 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T12:59:48.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:59:48.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:59:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3028301140,v1:127.0.0.1:6803/3028301140] [v2:127.0.0.1:6804/3028301140,v1:127.0.0.1:6805/3028301140] exists,up 5c72022f-cc7f-4f92-b554-436d9465a9d6 2024-03-09T12:59:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T12:59:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T12:59:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T12:59:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T12:59:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T12:59:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T12:59:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T12:59:49.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T12:59:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T12:59:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T12:59:49.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T12:59:49.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T12:59:49.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T12:59:49.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T12:59:49.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T12:59:49.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T12:59:49.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T12:59:49.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T12:59:49.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T12:59:49.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T12:59:49.635 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T12:59:49.635 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T12:59:49.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T12:59:49.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:59:49.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T12:59:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-09T12:59:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-09T12:59:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-09T12:59:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:59:49.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T12:59:50.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672987 2024-03-09T12:59:50.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672987 2024-03-09T12:59:50.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672987' 2024-03-09T12:59:50.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T12:59:50.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T12:59:50.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-09T12:59:50.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-09T12:59:50.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672987 2-64424509459' 2024-03-09T12:59:50.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:59:50.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-09T12:59:50.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:59:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T12:59:50.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-09T12:59:50.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:59:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-09T12:59:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-09T12:59:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 128849018882 2024-03-09T12:59:50.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:59:50.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-09T12:59:50.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:59:51.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T12:59:51.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:59:51.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-09T12:59:51.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T12:59:52.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T12:59:52.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T12:59:53.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-09T12:59:53.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:59:53.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672987 2024-03-09T12:59:53.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:59:53.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T12:59:53.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672987 2024-03-09T12:59:53.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:59:53.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672987 2024-03-09T12:59:53.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-03-09T12:59:53.015 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672987 2024-03-09T12:59:53.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T12:59:53.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672987 -lt 42949672987 2024-03-09T12:59:53.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T12:59:53.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-09T12:59:53.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T12:59:53.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T12:59:53.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-09T12:59:53.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T12:59:53.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-09T12:59:53.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-09T12:59:53.293 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509459 2024-03-09T12:59:53.293 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T12:59:53.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509459 2024-03-09T12:59:53.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T12:59:53.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:59:53.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:59:53.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T12:59:53.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T12:59:53.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T12:59:53.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T12:59:53.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T12:59:53.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T12:59:53.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T12:59:53.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T12:59:54.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T12:59:54.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T12:59:54.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T12:59:54.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T12:59:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T12:59:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T12:59:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T12:59:54.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2024-03-09T12:59:54.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T12:59:54.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T12:59:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T12:59:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T12:59:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T12:59:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T12:59:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T12:59:54.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T12:59:54.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T12:59:54.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T12:59:55.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T12:59:55.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T12:59:55.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T12:59:55.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T12:59:55.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T12:59:55.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T12:59:55.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T12:59:55.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T12:59:55.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T12:59:55.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T12:59:56.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T12:59:56.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T12:59:56.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T12:59:56.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T12:59:56.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T12:59:56.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T12:59:56.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T12:59:56.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T12:59:56.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T12:59:56.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T12:59:56.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T12:59:56.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T12:59:56.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T12:59:56.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T12:59:56.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T12:59:56.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T12:59:56.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T12:59:56.979 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T12:59:56.979 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T12:59:56.979 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T12:59:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T12:59:56.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T12:59:56.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T12:59:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T12:59:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T12:59:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T12:59:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T12:59:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:57.009+0000 7f23e5615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:59:57.009 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:57.012+0000 7f23e5615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:59:57.011 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:57.013+0000 7f23e5615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T12:59:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T12:59:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T12:59:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T12:59:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T12:59:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T12:59:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:59:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T12:59:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T12:59:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:59:57.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:59:57.529 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T12:59:57.531+0000 7f23e5615640 -1 Falling back to public interface 2024-03-09T12:59:57.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:59:58.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:59:58.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:59:58.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T12:59:58.536 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T12:59:58.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:59:58.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T12:59:58.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T12:59:59.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T12:59:59.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T12:59:59.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T12:59:59.821 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T12:59:59.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T12:59:59.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:00.047 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:00.049+0000 7f23e5615640 -1 osd.0 31 log_to_monitors true 2024-03-09T13:00:00.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:01.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:01.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:01.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:00:01.100 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:00:01.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:01.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:01.381 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 35 up_thru 0 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6802/342497983,v1:127.0.0.1:6803/342497983] [v2:127.0.0.1:6804/342497983,v1:127.0.0.1:6805/342497983] exists,up 5c72022f-cc7f-4f92-b554-436d9465a9d6 2024-03-09T13:00:01.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:00:01.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:00:01.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:00:01.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:00:01.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:00:01.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:00:01.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:00:01.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:00:01.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:00:01.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:00:01.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:00:01.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:00:01.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:00:01.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:00:01.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:00:01.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:00:01.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:00:01.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:00:01.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:00:01.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:00:01.760 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:00:01.760 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:00:01.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:00:01.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:00:01.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:00:01.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-09T13:00:01.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-09T13:00:01.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362' 2024-03-09T13:00:01.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:00:01.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:00:01.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672990 2024-03-09T13:00:01.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672990 2024-03-09T13:00:01.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672990' 2024-03-09T13:00:01.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:00:01.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:00:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-03-09T13:00:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-03-09T13:00:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-150323855362 1-42949672990 2-64424509463' 2024-03-09T13:00:02.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:00:02.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-150323855362 2024-03-09T13:00:02.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:00:02.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:00:02.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-150323855362 2024-03-09T13:00:02.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:00:02.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-09T13:00:02.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 150323855362' 2024-03-09T13:00:02.121 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 150323855362 2024-03-09T13:00:02.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:00:02.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-09T13:00:02.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:00:03.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:00:03.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:00:03.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 150323855362 2024-03-09T13:00:03.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:00:04.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:00:04.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:00:04.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-09T13:00:04.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:00:04.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672990 2024-03-09T13:00:04.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:00:04.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:00:04.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672990 2024-03-09T13:00:04.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:00:04.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672990 2024-03-09T13:00:04.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-03-09T13:00:04.957 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672990 2024-03-09T13:00:04.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:00:05.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672991 -lt 42949672990 2024-03-09T13:00:05.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:00:05.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-03-09T13:00:05.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:00:05.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:00:05.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-03-09T13:00:05.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:00:05.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-03-09T13:00:05.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-03-09T13:00:05.249 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509463 2024-03-09T13:00:05.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:00:05.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-03-09T13:00:05.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:00:05.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:00:05.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:00:05.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:00:05.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:00:05.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:00:05.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:00:05.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:00:05.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:00:05.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:00:05.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:00:06.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:00:06.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:00:06.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:00:06.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:00:06.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:00:06.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:00:06.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:00:06.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:00:06.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-09T13:00:06.476 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:59:36.504+0000 7f41e8105640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T13:00:06.476 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:59:36.504+0000 7f41e8105640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T13:00:06.476 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:59:36.504+0000 7f41e8105640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-09T13:00:06.476 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:59:36.504+0000 7f41e8105640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-09T13:00:06.476 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T12:59:36.504+0000 7f41e8105640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T13:00:06.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:00:06.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:00:06.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:00:06.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:00:06.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:00:06.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:00:06.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:00:06.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:00:06.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:00:06.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:00:06.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:00:06.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:00:06.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:00:06.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:00:06.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:00:06.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:00:06.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:00:06.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:00:06.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:00:06.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:00:06.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:00:06.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:00:06.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:00:06.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:00:06.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:00:06.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:00:06.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:00:06.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:00:06.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:00:06.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:00:06.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:00:06.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:00:06.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:00:06.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:00:06.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:00:06.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:00:06.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:00:06.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:00:06.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:00:06.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:00:06.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:00:06.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:00:06.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:00:06.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:00:06.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:00:06.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:00:06.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:00:06.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:00:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:00:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:00:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:00:06.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:00:06.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:00:06.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:00:06.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:00:06.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:00:06.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:00:06.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:00:06.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:00:06.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:00:06.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:00:06.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:00:06.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:00:06.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:00:06.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:00:06.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:00:06.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:00:06.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:00:06.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:00:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:00:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2024-03-09T13:00:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:539: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2024-03-09T13:00:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:540: TEST_auto_repair_bluestore_failed: local poolname=testpool 2024-03-09T13:00:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2024-03-09T13:00:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:00:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:00:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:00:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:00:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:00:06.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T13:00:07.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:00:07.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:00:07.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:00:07.079 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:00:07.079 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:00:07.079 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:00:07.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:00:07.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T13:00:07.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:00:07.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:00:07.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:00:07.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:00:07.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:00:07.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:00:07.272 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:00:07.272 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:00:07.272 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:00:07.273 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:00:07.273 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:00:07.273 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:00:07.273 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:00:07.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:00:07.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:00:07.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:00:07.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:00:07.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:00:07.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:00:07.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:00:07.353 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:00:07.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:00:07.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:00:07.354 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:00:07.354 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:00:07.354 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:00:07.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:00:07.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:00:07.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:00:07.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:544: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2024-03-09T13:00:07.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:00:07.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:00:07.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:00:07.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:00:07.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:00:07.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:00:08.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:00:08.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:00:08.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:00:08.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:00:08.241 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:00:08.242 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:00:08.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:00:08.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:00:08.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:00:08.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:00:08.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: seq 0 2 2024-03-09T13:00:08.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-09T13:00:08.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:00:08.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:00:08.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:00:08.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:00:08.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:00:08.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:00:08.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:00:08.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:00:08.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:00:08.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:00:08.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:00:08.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:00:08.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:00:08.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:00:08.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:00:08.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:00:08.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:00:08.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:00:08.275 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:00:08.275 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:00:08.275 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:00:08.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:00:08.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:00:08.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:00:08.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:00:08.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:00:08.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:00:08.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:00:08.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:00:08.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:00:08.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:00:08.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:00:08.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:00:08.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:00:08.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:00:08.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:00:08.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=84452a03-efb3-4069-aa59-4616c7588165 2024-03-09T13:00:08.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 84452a03-efb3-4069-aa59-4616c7588165' 2024-03-09T13:00:08.278 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 84452a03-efb3-4069-aa59-4616c7588165 2024-03-09T13:00:08.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:00:08.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBYXexlbkdxERAAdItG6ltI7OZpgcd8JeaETA== 2024-03-09T13:00:08.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBYXexlbkdxERAAdItG6ltI7OZpgcd8JeaETA=="}' 2024-03-09T13:00:08.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 84452a03-efb3-4069-aa59-4616c7588165 -i td/osd-scrub-repair/0/new.json 2024-03-09T13:00:08.496 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:00:08.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:00:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBYXexlbkdxERAAdItG6ltI7OZpgcd8JeaETA== --osd-uuid 84452a03-efb3-4069-aa59-4616c7588165 2024-03-09T13:00:08.536 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:08.538+0000 7f0b35b71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:00:08.546 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:08.548+0000 7f0b35b71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:00:08.547 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:08.549+0000 7f0b35b71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:00:08.548 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:08.550+0000 7f0b35b71640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:00:08.548 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:08.550+0000 7f0b35b71640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:00:08.548 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:08.550+0000 7f0b35b71640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:00:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:08.640+0000 7f0b35b71640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:00:14.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:00:14.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:00:14.170 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:00:14.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:00:14.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:00:14.550 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:00:14.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:00:14.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:00:14.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:00:14.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:00:14.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:00:14.578 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:14.581+0000 7f294ac11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:00:14.587 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:14.590+0000 7f294ac11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:00:14.588 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:14.591+0000 7f294ac11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:00:14.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:00:14.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:00:14.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:00:14.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:00:14.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:00:14.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:14.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:00:14.856 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:00:14.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:14.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:15.122 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:15.126+0000 7f294ac11640 -1 Falling back to public interface 2024-03-09T13:00:15.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:16.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:16.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:16.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:00:16.128 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:00:16.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:16.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:16.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:17.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:17.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:17.413 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:00:17.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:00:17.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:17.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:17.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:17.466+0000 7f294ac11640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:00:17.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:18.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:18.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:18.714 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:00:18.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:00:18.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:18.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:19.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:20.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:20.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:20.026 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:00:20.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:00:20.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:20.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:20.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:21.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:21.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:21.372 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:00:21.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:00:21.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:21.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:21.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:22.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:22.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:22.654 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:00:22.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:00:22.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:22.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:22.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:23.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:23.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:23.938 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:00:23.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:00:23.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:23.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:24.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:25.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:25.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:25.221 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:00:25.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:00:25.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:25.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:25.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:26.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:26.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:26.504 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:00:26.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:00:26.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:26.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:26.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:27.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:27.789 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:00:27.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:27.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:00:27.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:27.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:28.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:29.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:29.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:29.199 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:00:29.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:00:29.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:29.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:29.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:30.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:30.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:30.798 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:00:30.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:00:30.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:30.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:31.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:32.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:32.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:32.521 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:00:32.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:00:32.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:32.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:33.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:34.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:34.206 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:00:34.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:34.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:00:34.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:34.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:34.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:35.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:35.575 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:00:35.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:35.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:00:35.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:35.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:35.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:36.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:36.871 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:00:36.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:36.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:00:36.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:36.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:37.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:38.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:38.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:38.473 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:00:38.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:00:38.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:38.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:39.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:40.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:40.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:40.217 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:00:40.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:00:40.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:40.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:00:40.499 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3534449091,v1:127.0.0.1:6803/3534449091] [v2:127.0.0.1:6804/3534449091,v1:127.0.0.1:6805/3534449091] exists,up 84452a03-efb3-4069-aa59-4616c7588165 2024-03-09T13:00:40.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:00:40.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:00:40.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:00:40.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:00:40.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:00:40.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:00:40.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:00:40.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:00:40.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:00:40.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:00:40.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:00:40.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:00:40.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:00:40.506 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:00:40.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3fe82c6f-c98c-4d6e-8b02-f898b699fd76 2024-03-09T13:00:40.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3fe82c6f-c98c-4d6e-8b02-f898b699fd76' 2024-03-09T13:00:40.507 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 3fe82c6f-c98c-4d6e-8b02-f898b699fd76 2024-03-09T13:00:40.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:00:40.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB4XexlqTA3HxAAvumuPGCBSvhl2nFl7HXCNQ== 2024-03-09T13:00:40.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB4XexlqTA3HxAAvumuPGCBSvhl2nFl7HXCNQ=="}' 2024-03-09T13:00:40.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3fe82c6f-c98c-4d6e-8b02-f898b699fd76 -i td/osd-scrub-repair/1/new.json 2024-03-09T13:00:40.978 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:00:40.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:00:40.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQB4XexlqTA3HxAAvumuPGCBSvhl2nFl7HXCNQ== --osd-uuid 3fe82c6f-c98c-4d6e-8b02-f898b699fd76 2024-03-09T13:00:41.020 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:41.022+0000 7f5823c24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:00:41.054 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:41.057+0000 7f5823c24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:00:41.055 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:41.058+0000 7f5823c24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:00:41.056 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:41.058+0000 7f5823c24640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:00:41.056 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:41.058+0000 7f5823c24640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:00:41.056 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:41.058+0000 7f5823c24640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:00:41.138 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:41.140+0000 7f5823c24640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:00:47.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:00:47.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:00:47.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:00:47.297 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:00:47.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:00:47.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:00:47.653 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:00:47.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:00:47.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:00:47.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:00:47.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:00:47.681 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:47.683+0000 7fc6cbace640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:00:47.683 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:47.685+0000 7fc6cbace640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:00:47.683 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:47.686+0000 7fc6cbace640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:00:47.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:00:47.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:00:47.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:00:47.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:00:47.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:00:47.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:47.929 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:00:47.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:00:47.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:47.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:00:48.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:48.204 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:48.207+0000 7fc6cbace640 -1 Falling back to public interface 2024-03-09T13:00:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:00:49.199 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:00:49.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:49.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:00:49.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:50.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:50.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:50.477 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:00:50.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:00:50.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:50.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:00:50.695 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:00:50.697+0000 7fc6cbace640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:00:50.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:51.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:51.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:51.904 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:00:51.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:00:51.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:51.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:00:52.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:53.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:53.190 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:00:53.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:53.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:00:53.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:53.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:00:53.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:54.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:54.474 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:00:54.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:54.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:00:54.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:54.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:00:54.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:55.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:55.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:55.905 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:00:55.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:00:55.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:55.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:00:56.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:57.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:57.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:57.189 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:00:57.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:00:57.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:57.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:00:57.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:58.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:58.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:58.559 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:00:58.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:00:58.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:58.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:00:58.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:00:59.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:00:59.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:00:59.947 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:00:59.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:00:59.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:00:59.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:01:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:01.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:01.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:01.233 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:01:01.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:01:01.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:01.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:01:01.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:02.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:02.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:02.551 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:01:02.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:01:02.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:02.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:01:03.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:04.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:04.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:04.189 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:01:04.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:01:04.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:04.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:01:05.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:06.142 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:01:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:01:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:01:06.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:07.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:07.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:07.893 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:01:07.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:01:07.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:07.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:01:08.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:09.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:09.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:09.480 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:01:09.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:01:09.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:09.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:01:10.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:11.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:11.207 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:01:11.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:11.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:01:11.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:11.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:01:11.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:12.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:12.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:12.897 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:01:12.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:01:12.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:12.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:01:13.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:14.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:14.413 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:01:14.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:01:14.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:14.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:01:14.696 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4091111722,v1:127.0.0.1:6811/4091111722] [v2:127.0.0.1:6812/4091111722,v1:127.0.0.1:6813/4091111722] exists,up 3fe82c6f-c98c-4d6e-8b02-f898b699fd76 2024-03-09T13:01:14.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:548: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:549: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:01:14.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:01:14.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:01:14.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:01:14.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:01:14.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:01:14.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:01:14.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:01:14.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:01:14.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:01:14.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:01:14.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:01:14.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:01:14.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:01:14.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f2ac82a7-1b13-4870-942f-64fd9b202a0f 2024-03-09T13:01:14.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f2ac82a7-1b13-4870-942f-64fd9b202a0f' 2024-03-09T13:01:14.704 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 f2ac82a7-1b13-4870-942f-64fd9b202a0f 2024-03-09T13:01:14.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:01:14.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCaXexlzd/0KhAAlM+CkqN63LDPvIMzbmbf8w== 2024-03-09T13:01:14.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCaXexlzd/0KhAAlM+CkqN63LDPvIMzbmbf8w=="}' 2024-03-09T13:01:14.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f2ac82a7-1b13-4870-942f-64fd9b202a0f -i td/osd-scrub-repair/2/new.json 2024-03-09T13:01:15.129 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:01:15.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T13:01:15.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCaXexlzd/0KhAAlM+CkqN63LDPvIMzbmbf8w== --osd-uuid f2ac82a7-1b13-4870-942f-64fd9b202a0f 2024-03-09T13:01:15.171 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:15.174+0000 7f3fabe0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:01:15.206 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:15.209+0000 7f3fabe0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:01:15.206 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:15.210+0000 7f3fabe0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:01:15.207 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:15.210+0000 7f3fabe0f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:01:15.207 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:15.210+0000 7f3fabe0f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:01:15.207 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:15.210+0000 7f3fabe0f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:01:15.316 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:15.319+0000 7f3fabe0f640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T13:01:21.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T13:01:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:01:21.426 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T13:01:21.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T13:01:21.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:01:21.728 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T13:01:21.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T13:01:21.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:01:21.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:01:21.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:01:21.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:01:21.757 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:21.759+0000 7ffa47cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:01:21.758 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:21.761+0000 7ffa47cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:01:21.759 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:21.762+0000 7ffa47cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:01:22.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T13:01:22.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:01:22.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:01:22.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:01:22.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:01:22.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:22.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:01:22.003 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:01:22.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:22.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:22.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:22.286 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:22.288+0000 7ffa47cb5640 -1 Falling back to public interface 2024-03-09T13:01:23.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:23.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:23.273 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:01:23.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:01:23.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:23.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:23.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:24.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:24.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:24.556 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:01:24.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:01:24.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:24.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:24.733+0000 7ffa47cb5640 -1 osd.2 0 log_to_monitors true 2024-03-09T13:01:24.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:25.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:25.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:25.835 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:01:25.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:01:25.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:25.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:26.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:27.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:27.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:27.117 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:01:27.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:01:27.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:27.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:27.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:28.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:28.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:28.400 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:01:28.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:01:28.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:28.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:28.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:29.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:29.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:29.813 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:01:29.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:01:29.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:29.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:30.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:31.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:31.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:31.100 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:01:31.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:01:31.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:31.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:31.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:32.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:32.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:32.382 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:01:32.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:01:32.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:32.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:32.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:33.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:33.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:33.799 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:01:33.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:01:33.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:33.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:34.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:35.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:35.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:35.083 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:01:35.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:01:35.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:35.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:35.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:36.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:36.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:36.366 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:01:36.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:01:36.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:36.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:36.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:37.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:37.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:37.648 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:01:37.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:01:37.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:37.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:38.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:39.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:39.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:39.186 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:01:39.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:01:39.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:39.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:39.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:40.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:40.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:40.955 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:01:40.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:01:40.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:40.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:41.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:42.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:42.666 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:01:42.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:42.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:01:42.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:42.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:43.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:44.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:44.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:44.494 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:01:44.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:01:44.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:44.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:45.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:45.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:45.854 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:01:45.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:01:45.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:45.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:46.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:01:47.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:01:47.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:47.563 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:01:47.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:01:47.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:47.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:01:47.854 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/102407353,v1:127.0.0.1:6819/102407353] [v2:127.0.0.1:6820/102407353,v1:127.0.0.1:6821/102407353] exists,up f2ac82a7-1b13-4870-942f-64fd9b202a0f 2024-03-09T13:01:47.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:01:47.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:01:47.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:01:47.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2024-03-09T13:01:47.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-09T13:01:48.268 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'testpool' created 2024-03-09T13:01:48.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:01:49.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2024-03-09T13:01:49.639 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 1 size to 2 2024-03-09T13:01:49.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:554: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-03-09T13:01:49.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:01:49.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:01:49.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:01:49.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:01:49.652 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:01:49.652 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:01:49.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:01:49.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:01:49.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:01:49.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:01:49.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:01:49.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:01:49.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:01:49.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:01:49.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:01:50.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:01:50.254 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:01:50.254 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:01:50.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:01:50.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:01:50.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:01:50.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-03-09T13:01:50.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-03-09T13:01:50.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-03-09T13:01:50.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:01:50.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:01:50.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-03-09T13:01:50.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-03-09T13:01:50.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672970' 2024-03-09T13:01:50.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:01:50.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:01:50.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509443 2024-03-09T13:01:50.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509443 2024-03-09T13:01:50.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672970 2-64424509443' 2024-03-09T13:01:50.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:01:50.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-03-09T13:01:50.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:01:50.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:01:50.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-03-09T13:01:50.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:01:50.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-03-09T13:01:50.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-03-09T13:01:50.644 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836500 2024-03-09T13:01:50.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:01:50.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-09T13:01:50.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:01:51.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:01:51.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:01:52.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-09T13:01:52.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:01:53.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:01:53.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:01:53.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-03-09T13:01:53.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:01:53.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-03-09T13:01:53.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:01:53.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:01:53.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-03-09T13:01:53.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:01:53.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-03-09T13:01:53.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-03-09T13:01:53.482 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672970 2024-03-09T13:01:53.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:01:53.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-03-09T13:01:53.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:01:53.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509443 2024-03-09T13:01:53.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:01:53.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:01:53.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509443 2024-03-09T13:01:53.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:01:53.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509443 2024-03-09T13:01:53.759 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509443 2024-03-09T13:01:53.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2024-03-09T13:01:53.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:01:54.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509443 2024-03-09T13:01:54.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:01:54.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:01:54.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:01:54.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:01:54.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:01:54.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:01:54.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:01:54.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:01:54.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:01:54.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:01:54.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:01:54.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:01:54.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:01:54.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:01:54.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:01:55.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:01:55.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:01:55.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:01:55.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2024-03-09T13:01:55.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_failed: echo ABCDEF 2024-03-09T13:01:55.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: seq 1 10 2024-03-09T13:01:55.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T13:01:55.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:01:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T13:01:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:01:55.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T13:01:55.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:01:55.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T13:01:55.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:01:55.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T13:01:55.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:01:55.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T13:01:55.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:01:55.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T13:01:55.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:01:55.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T13:01:55.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:01:56.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T13:01:56.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:01:56.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:559: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2024-03-09T13:01:56.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:01:56.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-03-09T13:01:56.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T13:01:56.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:01:56.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T13:01:56.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:01:56.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:01:56.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:01:56.381 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:01:56.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T13:01:56.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:01:56.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T13:01:56.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:566: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-03-09T13:01:56.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:01:56.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:01:56.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:01:56.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:01:56.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-03-09T13:01:56.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:01:56.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:01:56.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:01:56.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:01:56.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:01:56.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:01:56.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:01:56.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:01:56.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:01:56.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:01:57.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:01:57.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-03-09T13:01:57.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:01:57.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:01:57.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:01:57.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:01:57.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:01:57.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-03-09T13:01:58.664 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-09T13:01:59.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:01:59.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:01:59.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:01:59.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:01:59.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:01:59.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:01:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:01:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:01:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:01:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:01:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:01:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:01:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:01:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:01:59.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:01:59.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:01:59.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:01:59.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:01:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:01:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:01:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:01:59.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:01:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:01:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:01:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:01:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:01:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:01:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:01:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:01:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:01:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:01:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:01:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:01:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:01:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:01:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:01:59.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:01:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:01:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:01:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:01:59.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:01:59.217 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:59.221+0000 7fbaf3819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:01:59.220 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:59.223+0000 7fbaf3819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:01:59.221 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:59.224+0000 7fbaf3819640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:01:59.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:01:59.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:01:59.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:01:59.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:01:59.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:01:59.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:01:59.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:01:59.471 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:01:59.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:01:59.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:01:59.737 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:01:59.740+0000 7fbaf3819640 -1 Falling back to public interface 2024-03-09T13:01:59.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:00.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:00.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:00.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:02:00.747 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:02:00.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:00.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:02:01.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:02.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:02.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:02.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:02:02.031 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:02:02.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:02.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:02:02.120 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:02.124+0000 7fbaf3819640 -1 osd.0 21 log_to_monitors true 2024-03-09T13:02:02.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:03.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:03.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:03.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:02:03.335 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:02:03.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:03.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:02:03.612 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3550157849,v1:127.0.0.1:6803/3550157849] [v2:127.0.0.1:6804/3550157849,v1:127.0.0.1:6805/3550157849] exists,up 84452a03-efb3-4069-aa59-4616c7588165 2024-03-09T13:02:03.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:02:03.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:02:03.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:02:03.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:02:03.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:02:03.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:02:03.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:02:03.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:02:03.613 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:02:03.613 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:02:03.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:02:03.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:02:03.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:02:03.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:02:03.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:02:03.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:02:03.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:02:03.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:02:03.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:02:03.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:02:03.969 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:02:03.969 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:02:03.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:02:03.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:03.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:02:04.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-09T13:02:04.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-09T13:02:04.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-09T13:02:04.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:04.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:02:04.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-03-09T13:02:04.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-03-09T13:02:04.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672973' 2024-03-09T13:02:04.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:04.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:02:04.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-03-09T13:02:04.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-03-09T13:02:04.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672973 2-64424509447' 2024-03-09T13:02:04.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:04.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-09T13:02:04.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:04.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:02:04.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-09T13:02:04.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:04.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-09T13:02:04.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-09T13:02:04.284 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 107374182402 2024-03-09T13:02:04.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:02:04.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-09T13:02:04.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:04.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-03-09T13:02:04.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:04.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:02:04.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-03-09T13:02:04.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:04.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-03-09T13:02:04.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-03-09T13:02:04.564 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672973 2024-03-09T13:02:04.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:02:04.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672974 -lt 42949672973 2024-03-09T13:02:04.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:04.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-03-09T13:02:04.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:04.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:02:04.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-03-09T13:02:04.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:04.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-03-09T13:02:04.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-03-09T13:02:04.844 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509447 2024-03-09T13:02:04.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:02:05.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-03-09T13:02:05.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:02:05.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:02:05.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:02:05.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:02:05.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:02:05.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:02:05.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:02:05.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:02:05.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:02:05.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:02:05.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:02:05.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:02:05.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:02:05.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:02:05.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:02:06.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:02:06.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:02:06.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:02:06.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2024-03-09T13:02:06.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T13:02:06.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:02:06.070 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T13:02:06.070 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:02:06.070 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:02:06.070 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:02:06.070 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:02:06.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T13:02:06.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:02:06.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T13:02:06.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:568: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-03-09T13:02:06.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:02:06.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:02:06.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:02:06.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:02:06.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-03-09T13:02:06.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:02:06.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:02:06.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:02:06.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:02:06.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:02:06.677 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:02:06.677 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:02:06.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:02:06.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:02:06.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:02:06.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:02:06.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-09T13:02:06.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:02:06.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:02:06.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:02:06.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:02:06.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:02:06.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-09T13:02:08.168 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:104778fc:::obj2:head# 2024-03-09T13:02:08.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:02:08.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:02:08.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:02:08.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:02:08.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:02:08.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:02:08.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:02:08.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:02:08.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:02:08.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:02:08.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:02:08.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:02:08.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:02:08.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:02:08.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:02:08.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:02:08.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:02:08.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:02:08.693 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:02:08.693 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:02:08.693 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:02:08.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:02:08.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:02:08.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:02:08.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:02:08.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:02:08.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:02:08.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:02:08.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:02:08.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:02:08.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:02:08.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:02:08.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:02:08.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:02:08.695 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:02:08.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:02:08.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:02:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:02:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:02:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:02:08.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:02:08.718 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:08.720+0000 7febdbc99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:02:08.721 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:08.723+0000 7febdbc99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:02:08.722 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:08.724+0000 7febdbc99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:02:08.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:02:08.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:02:08.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:02:08.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:02:08.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:02:08.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:08.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:02:08.975 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:02:08.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:08.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:02:09.244 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:09.247+0000 7febdbc99640 -1 Falling back to public interface 2024-03-09T13:02:09.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:10.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:10.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:10.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:02:10.248 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:02:10.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:10.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:02:10.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:11.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:11.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:11.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:02:11.553 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:02:11.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:11.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:02:11.565 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:11.568+0000 7febdbc99640 -1 osd.0 26 log_to_monitors true 2024-03-09T13:02:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:12.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:12.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:12.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:02:12.879 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:02:12.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:12.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:02:13.163 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3110111303,v1:127.0.0.1:6803/3110111303] [v2:127.0.0.1:6804/3110111303,v1:127.0.0.1:6805/3110111303] exists,up 84452a03-efb3-4069-aa59-4616c7588165 2024-03-09T13:02:13.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:02:13.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:02:13.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:02:13.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:02:13.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:02:13.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:02:13.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:02:13.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:02:13.164 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:02:13.164 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:02:13.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:02:13.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:02:13.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:02:13.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:02:13.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:02:13.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:02:13.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:02:13.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:02:13.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:02:13.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:02:13.520 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:02:13.520 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:02:13.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:02:13.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:13.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:02:13.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-09T13:02:13.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-09T13:02:13.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-09T13:02:13.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:13.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:02:13.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672976 2024-03-09T13:02:13.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672976 2024-03-09T13:02:13.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672976' 2024-03-09T13:02:13.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:13.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:02:13.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-03-09T13:02:13.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-03-09T13:02:13.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672976 2-64424509450' 2024-03-09T13:02:13.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:13.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-09T13:02:13.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:13.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:02:13.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-09T13:02:13.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:13.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-09T13:02:13.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-09T13:02:13.888 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 128849018882 2024-03-09T13:02:13.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:02:14.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-09T13:02:14.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:02:15.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:02:15.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:02:15.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-09T13:02:15.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:15.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672976 2024-03-09T13:02:15.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:15.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:02:15.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672976 2024-03-09T13:02:15.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:15.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672976 2024-03-09T13:02:15.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-03-09T13:02:15.450 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672976 2024-03-09T13:02:15.451 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:02:15.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672976 2024-03-09T13:02:15.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:15.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-03-09T13:02:15.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:15.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:02:15.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-03-09T13:02:15.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:15.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-03-09T13:02:15.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-03-09T13:02:15.726 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509450 2024-03-09T13:02:15.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:02:16.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-03-09T13:02:16.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:02:16.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:02:16.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:02:16.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:02:16.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:02:16.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:02:16.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:02:16.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:02:16.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:02:16.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:02:16.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:02:16.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:02:16.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:02:16.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:02:16.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:02:16.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:02:16.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:02:16.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:02:16.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-03-09T13:02:16.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:02:16.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:02:16.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:02:16.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:02:17.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-09T13:02:17.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:02:17.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:02:17.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:02:17.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:02:17.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-09T13:02:17.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:02:17.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:02:17.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:02:17.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:02:17.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:02:17.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:02:17.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:02:17.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:02:17.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:02:17.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:02:17.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:02:17.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-09T13:02:17.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:02:17.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:02:17.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:02:17.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:02:17.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:02:17.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-03-09T13:02:19.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:02:19.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:02:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:02:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:02:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:02:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:02:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:02:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:02:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:02:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:02:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:02:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:02:19.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:02:19.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:02:19.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:02:19.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:02:19.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:02:19.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:02:19.510 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:02:19.510 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:02:19.510 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:02:19.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:02:19.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:02:19.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:02:19.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:02:19.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:02:19.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:02:19.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:02:19.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:02:19.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:02:19.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:02:19.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:02:19.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:02:19.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:02:19.511 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:02:19.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:02:19.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:02:19.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:02:19.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:02:19.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:02:19.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:02:19.539 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:19.541+0000 7fa06f0f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:02:19.541 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:19.544+0000 7fa06f0f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:02:19.543 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:19.545+0000 7fa06f0f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:02:19.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:02:19.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:02:19.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:02:19.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:02:19.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:02:19.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:19.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:02:19.795 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:02:19.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:19.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:02:20.060 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:20.063+0000 7fa06f0f9640 -1 Falling back to public interface 2024-03-09T13:02:20.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:21.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:21.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:21.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:02:21.077 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:02:21.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:21.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:02:21.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:02:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:02:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:02:22.442 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:22.444+0000 7fa06f0f9640 -1 osd.1 31 log_to_monitors true 2024-03-09T13:02:22.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:23.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:23.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:23.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:02:23.682 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:02:23.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:23.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:02:23.959 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/117965463,v1:127.0.0.1:6811/117965463] [v2:127.0.0.1:6812/117965463,v1:127.0.0.1:6813/117965463] exists,up 3fe82c6f-c98c-4d6e-8b02-f898b699fd76 2024-03-09T13:02:23.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:02:23.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:02:23.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:02:23.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:02:23.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:02:23.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:02:23.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:02:23.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:02:23.960 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:02:23.960 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:02:23.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:02:23.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:02:23.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:02:24.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:02:24.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:02:24.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:02:24.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:02:24.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:02:24.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:02:24.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:02:24.315 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:02:24.315 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:02:24.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:02:24.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:24.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:02:24.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018885 2024-03-09T13:02:24.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018885 2024-03-09T13:02:24.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885' 2024-03-09T13:02:24.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:24.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:02:24.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-09T13:02:24.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-09T13:02:24.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362' 2024-03-09T13:02:24.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:24.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:02:24.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-09T13:02:24.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-09T13:02:24.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018885 1-150323855362 2-64424509453' 2024-03-09T13:02:24.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:24.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018885 2024-03-09T13:02:24.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:24.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:02:24.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018885 2024-03-09T13:02:24.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:24.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018885 2024-03-09T13:02:24.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2024-03-09T13:02:24.646 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 128849018885 2024-03-09T13:02:24.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:02:24.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018884 -lt 128849018885 2024-03-09T13:02:24.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:02:25.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:02:25.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:02:26.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018884 -lt 128849018885 2024-03-09T13:02:26.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:02:27.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:02:27.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:02:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018885 -lt 128849018885 2024-03-09T13:02:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:27.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-03-09T13:02:27.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:27.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:02:27.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-03-09T13:02:27.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:27.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-09T13:02:27.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-03-09T13:02:27.490 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 150323855362 2024-03-09T13:02:27.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:02:27.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-09T13:02:27.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:27.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-09T13:02:27.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:27.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:02:27.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-09T13:02:27.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:27.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-09T13:02:27.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-09T13:02:27.768 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509453 2024-03-09T13:02:27.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:02:28.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-03-09T13:02:28.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:02:28.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:02:28.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:02:28.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:02:28.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:02:28.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:02:28.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:02:28.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:02:28.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:02:28.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:02:28.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:02:28.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:02:28.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:02:28.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:02:28.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:02:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:02:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:02:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:02:29.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2024-03-09T13:02:29.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T13:02:29.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-09T13:02:29.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-09T13:02:29.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:02:29.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_failed: local pgid=1.0 2024-03-09T13:02:29.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2024-03-09T13:02:29.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:02:29.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T13:02:29.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T13:02:29.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:02:29.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_failed: local primary=1 2024-03-09T13:02:29.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2024-03-09T13:02:29.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:02:29.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:02:29.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:02:29.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:02:29.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:573: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2024-03-09T13:01:48.220534+0000 2024-03-09T13:02:29.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:574: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2024-03-09T13:02:29.934 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T13:02:29.934 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": true, 2024-03-09T13:02:29.935 INFO:tasks.workunit.client.0.smithi162.stdout: "must": false, 2024-03-09T13:02:29.935 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "2024-02-24T13:00:49.938245+0000" 2024-03-09T13:02:29.935 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T13:02:29.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:577: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2024-03-09T13:01:48.220534+0000 2024-03-09T13:02:29.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T13:02:29.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:01:48.220534+0000 2024-03-09T13:02:29.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:02:29.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:02:29.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:02:29.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:02:29.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:02:29.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:02:29.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:02:29.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:02:30.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:01:48.220534+0000 '>' 2024-03-09T13:01:48.220534+0000 2024-03-09T13:02:30.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:02:31.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:02:31.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:02:31.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:02:31.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:02:31.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:02:31.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:02:31.216 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:02:31.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:01:48.220534+0000 '>' 2024-03-09T13:01:48.220534+0000 2024-03-09T13:02:31.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:02:32.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:02:32.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:02:32.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:02:32.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:02:32.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:02:32.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:02:32.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:02:32.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:01:48.220534+0000 '>' 2024-03-09T13:01:48.220534+0000 2024-03-09T13:02:32.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:02:33.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:02:33.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:02:33.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:02:33.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:02:33.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:02:33.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:02:33.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:02:34.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:01:48.220534+0000 '>' 2024-03-09T13:01:48.220534+0000 2024-03-09T13:02:34.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:02:35.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:02:35.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:02:35.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:02:35.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:02:35.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:02:35.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:02:35.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:02:35.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:02:30.771195+0000 '>' 2024-03-09T13:01:48.220534+0000 2024-03-09T13:02:35.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:02:35.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_failed: wait_for_clean 2024-03-09T13:02:35.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:02:35.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:02:35.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:02:35.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:02:35.335 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:02:35.336 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:02:35.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:02:35.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:02:35.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:02:35.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:02:35.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:02:35.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:02:35.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:02:35.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:02:35.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:02:35.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:02:35.692 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:02:35.692 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:02:35.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:02:35.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:35.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:02:35.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018888 2024-03-09T13:02:35.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018888 2024-03-09T13:02:35.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888' 2024-03-09T13:02:35.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:35.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:02:35.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-03-09T13:02:35.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-03-09T13:02:35.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888 1-150323855365' 2024-03-09T13:02:35.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:35.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:02:36.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-09T13:02:36.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-09T13:02:36.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018888 1-150323855365 2-64424509456' 2024-03-09T13:02:36.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:36.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018888 2024-03-09T13:02:36.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:36.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:02:36.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018888 2024-03-09T13:02:36.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:36.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018888 2024-03-09T13:02:36.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018888' 2024-03-09T13:02:36.006 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 128849018888 2024-03-09T13:02:36.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:02:36.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018888 2024-03-09T13:02:36.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:02:37.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:02:37.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:02:37.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018888 -lt 128849018888 2024-03-09T13:02:37.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:37.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855365 2024-03-09T13:02:37.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:37.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:02:37.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855365 2024-03-09T13:02:37.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:37.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-03-09T13:02:37.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-03-09T13:02:37.571 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 150323855365 2024-03-09T13:02:37.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:02:37.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-03-09T13:02:37.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:37.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-09T13:02:37.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:37.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:02:37.847 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-09T13:02:37.847 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:37.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-09T13:02:37.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-09T13:02:37.849 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509456 2024-03-09T13:02:37.849 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:02:38.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-03-09T13:02:38.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:02:38.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:02:38.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:02:38.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:02:38.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:02:38.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:02:38.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:02:38.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:02:38.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:02:38.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:02:38.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:02:38.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:02:38.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:02:38.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:02:38.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:02:39.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:02:39.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:02:39.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:02:39.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-03-09T13:02:39.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:02:39.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:02:39.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:02:39.377 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:02:39.377 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:02:39.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:02:39.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:39.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:02:39.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018890 2024-03-09T13:02:39.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018890 2024-03-09T13:02:39.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890' 2024-03-09T13:02:39.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:39.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:02:39.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855367 2024-03-09T13:02:39.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855367 2024-03-09T13:02:39.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367' 2024-03-09T13:02:39.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:39.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:02:39.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-03-09T13:02:39.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-03-09T13:02:39.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018890 1-150323855367 2-64424509458' 2024-03-09T13:02:39.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:39.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018890 2024-03-09T13:02:39.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:39.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:02:39.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018890 2024-03-09T13:02:39.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:39.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018890 2024-03-09T13:02:39.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018890' 2024-03-09T13:02:39.719 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 128849018890 2024-03-09T13:02:39.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:02:39.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018890 2024-03-09T13:02:39.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:02:40.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:02:40.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:02:41.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018890 -lt 128849018890 2024-03-09T13:02:41.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:41.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855367 2024-03-09T13:02:41.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:41.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:02:41.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855367 2024-03-09T13:02:41.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:41.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855367 2024-03-09T13:02:41.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855367' 2024-03-09T13:02:41.279 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 150323855367 2024-03-09T13:02:41.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:02:41.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855367 -lt 150323855367 2024-03-09T13:02:41.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:41.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-03-09T13:02:41.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:41.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:02:41.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-03-09T13:02:41.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:41.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-03-09T13:02:41.557 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509458 2024-03-09T13:02:41.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-03-09T13:02:41.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:02:41.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-03-09T13:02:41.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-09T13:02:41.833 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T13:02:41.833 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T13:02:41.834 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-09T13:02:41.834 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-09T13:02:41.834 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T13:02:41.834 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.769+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T13:02:41.834 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.769+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-09T13:02:41.834 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.769+0000 7fa053072640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T13:02:41.834 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.769+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-09T13:02:41.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2024-03-09T13:02:41.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-09T13:02:42.093 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T13:02:42.093 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-03-09T13:02:30.771275+0000 21'10 36:95 [1,0] 1 [1,0] 1 21'10 2024-03-09T13:02:30.771195+0000 21'10 2024-03-09T13:02:30.771195+0000 0 1 periodic scrub scheduled @ 2024-03-10T13:02:30.771195+0000 10 0 2024-03-09T13:02:42.093 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T13:02:42.093 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T13:02:42.094 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T13:02:42.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-09T13:02:42.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2024-03-09T13:02:42.362 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T13:02:42.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2024-03-09T13:02:42.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T13:02:42.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-09T13:02:42.374 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-09T13:02:42.374 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:02:42.375 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T13:02:42.375 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T13:02:42.375 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:02:42.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T13:02:42.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-09T13:02:42.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T13:02:42.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2024-03-09T13:02:42.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:02:42.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:02:42.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:02:42.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:02:42.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2024-03-09T13:02:42.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:02:42.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:02:42.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:02:42.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:02:42.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:02:42.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:02:42.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:02:42.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:02:42.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:02:42.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:02:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:02:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2024-03-09T13:02:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:02:43.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:02:43.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:02:43.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:02:43.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:02:43.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2024-03-09T13:02:44.496 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:02:44.496 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:02:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:02:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:02:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:02:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:02:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:02:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:02:44.768 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:02:44.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:02:44.770 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:02:44.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:02:44.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:02:44.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:02:44.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:02:44.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:02:44.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:02:44.798 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:44.801+0000 7f43af78a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:02:44.801 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:44.803+0000 7f43af78a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:02:44.802 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:44.804+0000 7f43af78a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:02:45.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:02:45.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:02:45.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:02:45.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:02:45.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:02:45.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:45.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:02:45.049 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:02:45.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:45.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:02:45.317 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:45.321+0000 7f43af78a640 -1 Falling back to public interface 2024-03-09T13:02:45.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:46.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:46.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:46.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:02:46.325 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:02:46.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:46.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:02:46.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:47.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:47.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:47.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:02:47.608 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:02:47.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:47.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:02:47.731 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:47.734+0000 7f43af78a640 -1 osd.0 36 log_to_monitors true 2024-03-09T13:02:47.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:48.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:48.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:48.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:02:48.890 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:02:48.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:48.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:02:49.175 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 40 up_thru 32 down_at 37 last_clean_interval [30,36) [v2:127.0.0.1:6802/2500883366,v1:127.0.0.1:6803/2500883366] [v2:127.0.0.1:6804/2500883366,v1:127.0.0.1:6805/2500883366] exists,up 84452a03-efb3-4069-aa59-4616c7588165 2024-03-09T13:02:49.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:02:49.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:02:49.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:02:49.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:02:49.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:02:49.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:02:49.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:02:49.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:02:49.176 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:02:49.176 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:02:49.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:02:49.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:02:49.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:02:49.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:02:49.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:02:49.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:02:49.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:02:49.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:02:49.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:02:49.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:02:49.531 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:02:49.533 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:02:49.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:02:49.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:49.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:02:49.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-09T13:02:49.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-09T13:02:49.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842' 2024-03-09T13:02:49.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:49.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:02:49.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855370 2024-03-09T13:02:49.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855370 2024-03-09T13:02:49.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370' 2024-03-09T13:02:49.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:02:49.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:02:49.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-03-09T13:02:49.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-03-09T13:02:49.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691842 1-150323855370 2-64424509461' 2024-03-09T13:02:49.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:49.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691842 2024-03-09T13:02:49.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:49.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:02:49.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691842 2024-03-09T13:02:49.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:49.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-09T13:02:49.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2024-03-09T13:02:49.846 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 171798691842 2024-03-09T13:02:49.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:02:50.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 171798691842 2024-03-09T13:02:50.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:02:51.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:02:51.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:02:51.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-09T13:02:51.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:51.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855370 2024-03-09T13:02:51.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:51.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:02:51.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855370 2024-03-09T13:02:51.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:51.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855370 2024-03-09T13:02:51.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855370' 2024-03-09T13:02:51.402 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 150323855370 2024-03-09T13:02:51.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:02:51.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855370 -lt 150323855370 2024-03-09T13:02:51.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:02:51.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-03-09T13:02:51.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:02:51.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:02:51.677 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-03-09T13:02:51.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:02:51.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-03-09T13:02:51.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-03-09T13:02:51.679 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509461 2024-03-09T13:02:51.679 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:02:51.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509461 -lt 64424509461 2024-03-09T13:02:51.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:02:51.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:02:51.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:02:52.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:02:52.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:02:52.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:02:52.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:02:52.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:02:52.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:02:52.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:02:52.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:02:52.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:02:52.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:02:52.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:02:52.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:02:52.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:02:52.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:02:52.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:02:52.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2024-03-09T13:02:52.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:02:52.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-09T13:02:52.939 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T13:02:52.939 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T13:02:52.939 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-09T13:02:52.939 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-09T13:02:52.939 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T13:02:52.939 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.769+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T13:02:52.939 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.769+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-09T13:02:52.939 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.769+0000 7fa053072640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T13:02:52.939 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.769+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-09T13:02:52.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2024-03-09T13:02:52.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:02:52.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:02:52.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:02:52.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:02:53.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2024-03-09T13:02:53.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:02:53.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:02:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:02:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:02:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2024-03-09T13:02:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:02:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:02:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:02:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:02:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:02:53.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:02:53.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:02:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:02:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:02:53.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:02:53.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:02:53.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2024-03-09T13:02:53.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:02:53.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:02:53.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:02:53.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:02:53.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:02:53.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2024-03-09T13:02:54.764 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2024-03-09T13:02:54.964 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:104778fc:::obj2:head# 2024-03-09T13:02:55.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:02:55.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:02:55.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:02:55.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:02:55.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:02:55.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:02:55.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:02:55.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:02:55.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:02:55.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:02:55.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:02:55.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:02:55.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:02:55.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:02:55.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:02:55.295 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:02:55.295 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:02:55.295 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:02:55.295 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:02:55.296 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:02:55.296 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:02:55.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:02:55.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:02:55.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:02:55.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:02:55.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:02:55.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:02:55.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:02:55.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:02:55.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:02:55.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:02:55.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:02:55.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:02:55.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:02:55.298 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:02:55.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:02:55.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:02:55.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:02:55.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:02:55.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:02:55.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:02:55.318 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:55.320+0000 7f29cb16b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:02:55.320 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:55.323+0000 7f29cb16b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:02:55.321 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:55.324+0000 7f29cb16b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:02:55.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:02:55.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:02:55.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:02:55.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:02:55.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:02:55.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:55.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:02:55.572 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:02:55.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:55.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:02:55.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:55.845 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:55.848+0000 7f29cb16b640 -1 Falling back to public interface 2024-03-09T13:02:56.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:56.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:56.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:02:56.846 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:02:56.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:56.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:02:57.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:58.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:58.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:58.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:02:58.128 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:02:58.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:58.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:02:58.145 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:02:58.147+0000 7f29cb16b640 -1 osd.1 41 log_to_monitors true 2024-03-09T13:02:58.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:02:59.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:02:59.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:02:59.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:02:59.441 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:02:59.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:02:59.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:02:59.720 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [35,41) [v2:127.0.0.1:6810/388502915,v1:127.0.0.1:6811/388502915] [v2:127.0.0.1:6812/388502915,v1:127.0.0.1:6813/388502915] exists,up 3fe82c6f-c98c-4d6e-8b02-f898b699fd76 2024-03-09T13:02:59.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:02:59.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:02:59.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:02:59.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:02:59.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:02:59.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:02:59.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:02:59.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:02:59.720 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:02:59.720 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:02:59.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:02:59.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:02:59.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:02:59.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:02:59.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:02:59.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:02:59.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:02:59.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:02:59.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:03:00.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:03:00.077 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:03:00.077 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:03:00.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:03:00.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:03:00.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:03:00.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-03-09T13:03:00.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-03-09T13:03:00.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845' 2024-03-09T13:03:00.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:03:00.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:03:00.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528322 2024-03-09T13:03:00.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528322 2024-03-09T13:03:00.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322' 2024-03-09T13:03:00.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:03:00.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:03:00.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-03-09T13:03:00.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-03-09T13:03:00.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691845 1-193273528322 2-64424509464' 2024-03-09T13:03:00.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:03:00.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691845 2024-03-09T13:03:00.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:03:00.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:03:00.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691845 2024-03-09T13:03:00.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:03:00.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-03-09T13:03:00.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691845' 2024-03-09T13:03:00.397 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 171798691845 2024-03-09T13:03:00.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:03:00.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-03-09T13:03:00.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:03:00.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528322 2024-03-09T13:03:00.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:03:00.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:03:00.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528322 2024-03-09T13:03:00.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:03:00.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528322 2024-03-09T13:03:00.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2024-03-09T13:03:00.709 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 193273528322 2024-03-09T13:03:00.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:03:00.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528322 -lt 193273528322 2024-03-09T13:03:00.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:03:00.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-03-09T13:03:00.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:03:00.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:03:00.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-03-09T13:03:00.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:03:00.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-03-09T13:03:00.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-03-09T13:03:00.986 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509464 2024-03-09T13:03:00.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:03:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509464 2024-03-09T13:03:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:03:02.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:03:02.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:03:02.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509464 2024-03-09T13:03:02.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:03:02.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:03:02.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:03:02.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:03:02.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:03:02.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:03:02.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:03:02.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:03:02.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:03:02.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:03:02.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:03:03.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:03:03.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:03:03.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:03:03.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:03:03.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:03:03.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:03:03.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:03:03.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_failed: repair 1.0 2024-03-09T13:03:03.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-09T13:03:03.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-09T13:03:03.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:03:03.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:03:03.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:03:03.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:03:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:02:30.771195+0000 2024-03-09T13:03:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-09T13:03:04.041 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 1.0 on osd.1 to repair 2024-03-09T13:03:04.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-09T13:02:30.771195+0000 2024-03-09T13:03:04.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T13:03:04.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:02:30.771195+0000 2024-03-09T13:03:04.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:03:04.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:03:04.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:03:04.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:03:04.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:03:04.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:03:04.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:03:04.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:03:04.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:02:30.771195+0000 '>' 2024-03-09T13:02:30.771195+0000 2024-03-09T13:03:04.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:03:05.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:03:05.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:03:05.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:03:05.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:03:05.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:03:05.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:03:05.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:03:05.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:02:30.771195+0000 '>' 2024-03-09T13:02:30.771195+0000 2024-03-09T13:03:05.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:03:06.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:03:06.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:03:06.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:03:06.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:03:06.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:03:06.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:03:06.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:03:06.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:02:30.771195+0000 '>' 2024-03-09T13:02:30.771195+0000 2024-03-09T13:03:06.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:03:07.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:03:07.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:03:07.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:03:07.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:03:07.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:03:07.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:03:07.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:03:08.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:02:30.771195+0000 '>' 2024-03-09T13:02:30.771195+0000 2024-03-09T13:03:08.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:03:09.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:03:09.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:03:09.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:03:09.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:03:09.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:03:09.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:03:09.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:03:09.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:02:30.771195+0000 '>' 2024-03-09T13:02:30.771195+0000 2024-03-09T13:03:09.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:03:10.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:03:10.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:03:10.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:03:10.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:03:10.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:03:10.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:03:10.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:03:10.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:03:04.198353+0000 '>' 2024-03-09T13:02:30.771195+0000 2024-03-09T13:03:10.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:03:10.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:595: TEST_auto_repair_bluestore_failed: sleep 2 2024-03-09T13:03:12.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:597: TEST_auto_repair_bluestore_failed: flush_pg_stats 2024-03-09T13:03:12.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:03:12.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:03:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:03:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:03:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:03:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:03:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:03:13.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:03:13.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691849 2024-03-09T13:03:13.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691849 2024-03-09T13:03:13.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849' 2024-03-09T13:03:13.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:03:13.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:03:13.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528325 2024-03-09T13:03:13.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528325 2024-03-09T13:03:13.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849 1-193273528325' 2024-03-09T13:03:13.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:03:13.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:03:13.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509468 2024-03-09T13:03:13.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509468 2024-03-09T13:03:13.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691849 1-193273528325 2-64424509468' 2024-03-09T13:03:13.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:03:13.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691849 2024-03-09T13:03:13.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:03:13.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:03:13.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691849 2024-03-09T13:03:13.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:03:13.350 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 171798691849 2024-03-09T13:03:13.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691849 2024-03-09T13:03:13.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691849' 2024-03-09T13:03:13.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:03:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691847 -lt 171798691849 2024-03-09T13:03:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:03:14.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:03:14.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:03:14.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691849 -lt 171798691849 2024-03-09T13:03:14.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:03:14.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-193273528325 2024-03-09T13:03:14.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:03:14.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:03:14.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-193273528325 2024-03-09T13:03:14.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:03:14.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528325 2024-03-09T13:03:14.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 193273528325' 2024-03-09T13:03:14.928 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 193273528325 2024-03-09T13:03:14.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:03:15.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528326 -lt 193273528325 2024-03-09T13:03:15.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:03:15.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509468 2024-03-09T13:03:15.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:03:15.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:03:15.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509468 2024-03-09T13:03:15.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:03:15.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509468 2024-03-09T13:03:15.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-03-09T13:03:15.205 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509468 2024-03-09T13:03:15.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:03:15.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509468 -lt 64424509468 2024-03-09T13:03:15.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-09T13:03:15.761 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T13:03:15.761 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2024-03-09T13:03:04.198441+0000 21'10 46:132 [1,0] 1 [1,0] 1 21'10 2024-03-09T13:03:04.198353+0000 21'10 2024-03-09T13:03:04.198353+0000 0 1 periodic scrub scheduled @ 2024-03-10T13:03:04.198353+0000 9 0 2024-03-09T13:03:15.761 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T13:03:15.761 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T13:03:15.762 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T13:03:15.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2024-03-09T13:03:15.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2024-03-09T13:03:16.027 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T13:03:16.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:600: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-09T13:03:16.041 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T13:03:16.041 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T13:03:16.041 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 15 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2024-03-09T13:03:16.041 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 20 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish Current 'required': 0 Planned 'req_scrub': 0 2024-03-09T13:03:16.041 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.475+0000 7fa053072640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T13:03:16.041 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.769+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] planned CHECK_REPAIR MUST_DEEP_SCRUB MUST_SCRUB] scrubber: scrub_finish before flags: CHECK_REPAIR REQ_SCRUB. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T13:03:16.041 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.769+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-09T13:03:16.041 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.769+0000 7fa053072640 19 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: CHECK_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T13:03:16.041 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:02:30.769+0000 7fa053072640 10 osd.1 pg_epoch: 36 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=35/36 n=10 ec=17/17 lis/c=35/35 les/c/f=36/36/0 sis=35) [1,0] r=0 lpr=35 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair [ 1.0: REQ_SCRUB ] ] scrubber: scrub_finish 3 error(s) still present after re-scrub 2024-03-09T13:03:16.041 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:03:04.196+0000 7f29af0e4640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] MUST_REPAIR planned AUTO_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] scrubber: scrub_finish before flags: AUTO_REPAIR REQ_SCRUB. repair state: repair. deep_scrub_on_error: 0 2024-03-09T13:03:16.041 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:03:04.196+0000 7f29af0e4640 10 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=10 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T13:03:16.042 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:03:04.196+0000 7f29af0e4640 15 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-09T13:03:16.042 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:03:04.196+0000 7f29af0e4640 20 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish All may be fixed 2024-03-09T13:03:16.042 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:03:04.196+0000 7f29af0e4640 19 osd.1 pg_epoch: 46 pg[1.0( v 21'10 (0'0,21'10] local-lis/les=45/46 n=9 ec=17/17 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=21'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair [ 1.0: AUTO_REPAIR REQ_SCRUB ] ] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T13:03:16.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:03:16.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:03:16.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:03:16.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:03:16.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:03:16.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:03:16.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:03:16.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:03:16.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:03:16.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:03:16.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:03:16.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:03:16.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:03:16.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:03:16.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:03:16.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:03:16.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:03:16.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:03:16.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:03:16.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:03:16.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:03:16.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:03:16.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:03:16.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:03:16.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:03:16.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:03:16.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:03:16.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:03:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:03:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:03:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:03:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:03:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:03:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:03:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:03:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:03:16.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:03:16.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:03:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:03:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:03:16.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:03:16.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:03:16.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:03:16.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:03:16.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:03:16.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:03:16.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:03:16.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:03:16.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:03:16.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:03:16.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:03:16.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:03:16.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:03:16.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:03:16.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:03:16.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:03:16.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:03:16.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:03:16.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:03:16.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:03:16.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:03:16.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:03:16.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:03:16.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:03:16.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:03:16.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:03:16.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:03:16.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:03:16.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:03:16.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:03:16.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2024-03-09T13:03:16.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:604: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2024-03-09T13:03:16.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:605: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2024-03-09T13:03:16.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2024-03-09T13:03:16.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:03:16.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:03:16.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:03:16.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:03:16.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:03:16.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T13:03:16.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:03:16.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:03:16.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:03:16.647 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:03:16.647 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:03:16.648 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:03:16.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:03:16.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T13:03:16.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:03:16.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:03:16.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:03:16.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:03:16.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:03:16.849 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:03:16.849 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:03:16.849 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:03:16.849 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:03:16.850 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:03:16.850 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:03:16.850 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:03:16.850 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:03:16.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:03:16.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:03:16.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:03:16.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:03:16.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:03:16.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:03:16.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:03:16.933 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:03:16.933 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:03:16.933 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:03:16.933 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:03:16.933 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:03:16.933 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:03:16.933 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:03:16.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:03:16.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:03:17.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:609: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2024-03-09T13:03:17.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:03:17.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:03:17.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:03:17.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:03:17.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:03:17.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:03:17.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:03:17.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:03:17.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:03:17.824 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:03:17.824 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:03:17.824 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:03:17.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:03:17.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:03:17.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:03:17.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:03:17.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2024-03-09T13:03:17.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-09T13:03:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:03:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:03:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:03:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:03:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:03:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:03:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:03:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:03:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:03:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:03:17.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:03:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:03:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:03:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:03:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:03:17.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:03:17.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:03:17.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:03:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:03:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:03:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:03:17.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:03:17.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:03:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:03:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:03:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:03:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:03:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:03:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:03:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:03:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:03:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:03:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:03:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:03:17.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:03:17.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:03:17.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=468df717-fddc-44ba-a73d-6658db3087b5 2024-03-09T13:03:17.861 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 468df717-fddc-44ba-a73d-6658db3087b5 2024-03-09T13:03:17.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 468df717-fddc-44ba-a73d-6658db3087b5' 2024-03-09T13:03:17.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:03:17.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVXuxliBlhNBAAUY31pI4CtnrU+A2aTuRCxA== 2024-03-09T13:03:17.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVXuxliBlhNBAAUY31pI4CtnrU+A2aTuRCxA=="}' 2024-03-09T13:03:17.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 468df717-fddc-44ba-a73d-6658db3087b5 -i td/osd-scrub-repair/0/new.json 2024-03-09T13:03:18.080 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:03:18.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:03:18.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAVXuxliBlhNBAAUY31pI4CtnrU+A2aTuRCxA== --osd-uuid 468df717-fddc-44ba-a73d-6658db3087b5 2024-03-09T13:03:18.120 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:18.124+0000 7f2f23916640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:03:18.130 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:18.134+0000 7f2f23916640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:03:18.131 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:18.135+0000 7f2f23916640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:03:18.132 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:18.135+0000 7f2f23916640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:03:18.132 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:18.135+0000 7f2f23916640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:03:18.132 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:18.135+0000 7f2f23916640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:03:18.214 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:18.217+0000 7f2f23916640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:03:23.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:03:23.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:03:23.712 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:03:23.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:03:23.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:03:24.099 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:03:24.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:03:24.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:03:24.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:03:24.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:03:24.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:03:24.128 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:24.131+0000 7fe1e7811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:03:24.144 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:24.147+0000 7fe1e7811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:03:24.145 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:24.148+0000 7fe1e7811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:03:24.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:03:24.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:03:24.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:03:24.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:03:24.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:03:24.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:24.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:03:24.402 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:03:24.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:24.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:24.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:24.686 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:24.689+0000 7fe1e7811640 -1 Falling back to public interface 2024-03-09T13:03:25.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:25.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:25.678 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:03:25.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:03:25.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:25.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:25.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:26.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:26.955 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:03:26.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:26.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:03:26.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:26.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:27.043 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:27.046+0000 7fe1e7811640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:03:27.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:28.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:28.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:28.271 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:03:28.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:03:28.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:28.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:28.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:29.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:29.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:29.548 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:03:29.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:03:29.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:29.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:29.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:30.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:30.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:30.896 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:03:30.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:03:30.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:30.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:31.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:32.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:32.180 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:03:32.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:32.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:03:32.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:32.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:32.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:33.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:33.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:33.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:03:33.465 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:03:33.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:33.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:33.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:34.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:34.792 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:03:34.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:34.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:03:34.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:34.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:35.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:36.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:36.075 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:03:36.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:36.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:03:36.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:36.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:36.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:37.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:37.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:37.354 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:03:37.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:03:37.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:37.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:37.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:38.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:38.696 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:03:38.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:38.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:03:38.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:38.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:39.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:40.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:40.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:40.198 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:03:40.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:03:40.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:40.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:40.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:41.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:41.641 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:03:41.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:41.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:03:41.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:41.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:42.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:43.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:43.370 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:03:43.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:43.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:03:43.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:43.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:43.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:44.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:44.805 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:03:44.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:44.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:03:44.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:44.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:45.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:46.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:46.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:46.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:03:46.167 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:03:46.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:46.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:46.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:47.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:47.830 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:03:47.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:47.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:03:47.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:47.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:48.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:49.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:49.120 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:03:49.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:49.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:03:49.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:49.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:03:49.398 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2635551744,v1:127.0.0.1:6803/2635551744] [v2:127.0.0.1:6804/2635551744,v1:127.0.0.1:6805/2635551744] exists,up 468df717-fddc-44ba-a73d-6658db3087b5 2024-03-09T13:03:49.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:03:49.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:03:49.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:03:49.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-09T13:03:49.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:03:49.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:03:49.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:03:49.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:03:49.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:03:49.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:03:49.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:03:49.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:03:49.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:03:49.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:03:49.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:03:49.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:03:49.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:03:49.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:03:49.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:03:49.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:03:49.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:03:49.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:03:49.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:03:49.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:03:49.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:03:49.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:03:49.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1b03a655-6f51-4fb8-9a0c-9f1b94afc80e 2024-03-09T13:03:49.405 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 1b03a655-6f51-4fb8-9a0c-9f1b94afc80e 2024-03-09T13:03:49.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1b03a655-6f51-4fb8-9a0c-9f1b94afc80e' 2024-03-09T13:03:49.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:03:49.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1XuxlmbcmGRAAccwPod0Ora/i/WzT0Uf7tA== 2024-03-09T13:03:49.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1XuxlmbcmGRAAccwPod0Ora/i/WzT0Uf7tA=="}' 2024-03-09T13:03:49.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1b03a655-6f51-4fb8-9a0c-9f1b94afc80e -i td/osd-scrub-repair/1/new.json 2024-03-09T13:03:49.771 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:03:49.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:03:49.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA1XuxlmbcmGRAAccwPod0Ora/i/WzT0Uf7tA== --osd-uuid 1b03a655-6f51-4fb8-9a0c-9f1b94afc80e 2024-03-09T13:03:49.814 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:49.817+0000 7fb192b06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:03:49.856 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:49.858+0000 7fb192b06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:03:49.857 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:49.859+0000 7fb192b06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:03:49.857 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:49.860+0000 7fb192b06640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:03:49.857 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:49.860+0000 7fb192b06640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:03:49.857 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:49.860+0000 7fb192b06640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:03:49.930 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:49.933+0000 7fb192b06640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:03:55.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:03:55.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:03:55.662 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:03:55.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:03:55.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:03:56.023 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:03:56.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:03:56.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:03:56.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:03:56.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:03:56.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:03:56.051 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:56.054+0000 7f2ea060d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:03:56.053 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:56.056+0000 7f2ea060d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:03:56.054 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:56.057+0000 7f2ea060d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:03:56.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:03:56.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:03:56.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:03:56.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:03:56.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:03:56.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:56.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:03:56.298 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:03:56.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:56.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:03:56.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:56.579 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:56.582+0000 7f2ea060d640 -1 Falling back to public interface 2024-03-09T13:03:57.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:57.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:57.571 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:03:57.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:03:57.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:57.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:03:57.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:03:58.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:03:58.854 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:03:58.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:03:58.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:03:58.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:03:58.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:03:58.879 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:03:58.881+0000 7f2ea060d640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:03:59.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:00.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:00.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:00.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:04:00.213 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:04:00.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:00.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:00.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:01.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:01.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:01.542 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:04:01.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:04:01.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:01.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:01.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:02.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:02.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:02.966 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:04:02.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:04:02.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:02.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:03.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:04.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:04.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:04.249 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:04:04.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:04:04.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:04.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:04.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:05.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:05.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:05.712 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:04:05.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:04:05.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:05.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:05.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:06.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:06.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:06.995 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:04:06.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:04:06.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:06.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:07.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:08.279 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:04:08.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:04:08.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:08.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:08.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:09.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:09.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:09.577 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:04:09.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:04:09.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:09.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:09.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:10.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:10.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:10.891 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:04:10.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:04:10.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:10.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:11.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:12.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:12.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:12.415 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:04:12.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:04:12.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:12.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:12.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:13.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:13.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:13.995 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:04:13.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:04:13.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:13.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:14.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:15.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:15.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:15.740 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:04:15.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:04:15.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:15.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:16.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:17.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:17.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:17.376 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:04:17.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:04:17.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:17.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:17.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:18.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:18.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:18.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:04:18.840 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:04:18.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:18.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:19.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:20.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:20.128 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:04:20.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:20.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:04:20.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:20.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:20.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:21.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:21.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:21.890 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:04:21.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:04:21.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:21.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:04:22.175 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4103546855,v1:127.0.0.1:6811/4103546855] [v2:127.0.0.1:6812/4103546855,v1:127.0.0.1:6813/4103546855] exists,up 1b03a655-6f51-4fb8-9a0c-9f1b94afc80e 2024-03-09T13:04:22.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:04:22.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:04:22.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:04:22.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2024-03-09T13:04:22.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:614: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:04:22.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:04:22.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:04:22.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T13:04:22.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:04:22.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:04:22.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:04:22.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:04:22.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:04:22.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:04:22.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:04:22.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:04:22.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:04:22.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:04:22.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:04:22.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:04:22.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:04:22.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:04:22.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:04:22.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:04:22.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:04:22.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:04:22.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3616030f-8f67-46f3-a0b0-48b4d71ef0d8 2024-03-09T13:04:22.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3616030f-8f67-46f3-a0b0-48b4d71ef0d8' 2024-03-09T13:04:22.183 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 3616030f-8f67-46f3-a0b0-48b4d71ef0d8 2024-03-09T13:04:22.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:04:22.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBWXuxl1z/nCxAAXFW3ZsULt+b+jtXVlhztyQ== 2024-03-09T13:04:22.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBWXuxl1z/nCxAAXFW3ZsULt+b+jtXVlhztyQ=="}' 2024-03-09T13:04:22.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3616030f-8f67-46f3-a0b0-48b4d71ef0d8 -i td/osd-scrub-repair/2/new.json 2024-03-09T13:04:22.602 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:04:22.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T13:04:22.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBWXuxl1z/nCxAAXFW3ZsULt+b+jtXVlhztyQ== --osd-uuid 3616030f-8f67-46f3-a0b0-48b4d71ef0d8 2024-03-09T13:04:22.644 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:04:22.647+0000 7feb94fa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:04:22.686 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:04:22.690+0000 7feb94fa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:04:22.687 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:04:22.691+0000 7feb94fa8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:04:22.688 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:04:22.691+0000 7feb94fa8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:04:22.688 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:04:22.691+0000 7feb94fa8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:04:22.688 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:04:22.691+0000 7feb94fa8640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:04:22.761 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:04:22.764+0000 7feb94fa8640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T13:04:28.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T13:04:28.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:04:28.386 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T13:04:28.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T13:04:28.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:04:28.737 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T13:04:28.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T13:04:28.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:04:28.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:04:28.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:04:28.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:04:28.765 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:04:28.767+0000 7f388e40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:04:28.766 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:04:28.769+0000 7f388e40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:04:28.767 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:04:28.770+0000 7f388e40d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:04:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T13:04:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:04:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:04:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:04:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:04:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:04:29.014 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:04:29.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:29.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:29.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:29.294 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:04:29.297+0000 7f388e40d640 -1 Falling back to public interface 2024-03-09T13:04:30.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:30.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:30.285 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:04:30.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:04:30.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:30.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:30.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:31.560 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:04:31.562+0000 7f388e40d640 -1 osd.2 0 log_to_monitors true 2024-03-09T13:04:31.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:31.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:31.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:04:31.568 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:04:31.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:31.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:31.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:32.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:04:32.870 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:04:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:32.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:33.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:34.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:34.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:34.183 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:04:34.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:04:34.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:34.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:34.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:35.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:35.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:35.468 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:04:35.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:04:35.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:35.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:35.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:36.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:36.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:36.751 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:04:36.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:04:36.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:36.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:37.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:38.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:38.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:38.039 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:04:38.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:04:38.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:38.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:38.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:39.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:39.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:39.575 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:04:39.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:04:39.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:39.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:39.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:40.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:40.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:40.857 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:04:40.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:04:40.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:40.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:41.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:42.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:42.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:42.176 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:04:42.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:04:42.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:42.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:42.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:43.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:43.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:43.511 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:04:43.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:04:43.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:43.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:44.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:45.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:45.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:45.038 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:04:45.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:04:45.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:45.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:45.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:46.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:46.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:46.595 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:04:46.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:04:46.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:46.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:47.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:48.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:48.227 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:04:48.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:48.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:04:48.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:48.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:48.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:49.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:49.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:49.536 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:04:49.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:04:49.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:49.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:50.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:51.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:51.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:51.004 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:04:51.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:04:51.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:51.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:51.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:52.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:52.758 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:04:52.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:52.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:04:52.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:52.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:53.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:04:54.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:04:54.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:04:54.316 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:04:54.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:04:54.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:04:54.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:04:54.598 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1465856257,v1:127.0.0.1:6819/1465856257] [v2:127.0.0.1:6820/1465856257,v1:127.0.0.1:6821/1465856257] exists,up 3616030f-8f67-46f3-a0b0-48b4d71ef0d8 2024-03-09T13:04:54.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:04:54.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:04:54.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:04:54.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2024-03-09T13:04:54.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-09T13:04:55.120 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'testpool' created 2024-03-09T13:04:55.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:04:56.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2024-03-09T13:04:56.511 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 1 size to 2 2024-03-09T13:04:56.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:619: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-03-09T13:04:56.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:04:56.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:04:56.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:04:56.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:04:56.523 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:04:56.523 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:04:56.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:04:56.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:04:56.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:04:56.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:04:56.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:04:56.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:04:56.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:04:56.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:04:56.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:04:56.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:04:56.880 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:04:56.881 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:04:56.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:04:56.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:04:56.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:04:57.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-03-09T13:04:57.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-03-09T13:04:57.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-03-09T13:04:57.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:04:57.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:04:57.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-09T13:04:57.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-09T13:04:57.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672975' 2024-03-09T13:04:57.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:04:57.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:04:57.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-03-09T13:04:57.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-03-09T13:04:57.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672975 2-64424509445' 2024-03-09T13:04:57.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:04:57.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-03-09T13:04:57.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:04:57.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:04:57.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-03-09T13:04:57.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:04:57.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-03-09T13:04:57.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-03-09T13:04:57.341 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836500 2024-03-09T13:04:57.341 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:04:57.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-09T13:04:57.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:04:58.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:04:58.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:04:58.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-03-09T13:04:58.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:04:58.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-09T13:04:58.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:04:58.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:04:58.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-09T13:04:58.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:04:58.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-09T13:04:58.900 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672975 2024-03-09T13:04:58.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-09T13:04:58.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:04:59.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672976 -lt 42949672975 2024-03-09T13:04:59.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:04:59.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-03-09T13:04:59.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:04:59.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:04:59.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-03-09T13:04:59.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:04:59.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-03-09T13:04:59.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-03-09T13:04:59.174 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509445 2024-03-09T13:04:59.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:04:59.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509445 -lt 64424509445 2024-03-09T13:04:59.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:04:59.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:04:59.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:04:59.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:04:59.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:04:59.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:04:59.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:04:59.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:04:59.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:04:59.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:04:59.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:05:00.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:05:00.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:05:00.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:05:00.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:05:00.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:05:00.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:05:00.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:05:00.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:622: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2024-03-09T13:05:00.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:623: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2024-03-09T13:05:00.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2024-03-09T13:05:00.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T13:05:00.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:05:00.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T13:05:00.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:05:00.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T13:05:00.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:05:00.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T13:05:00.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:05:00.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T13:05:00.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:05:01.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T13:05:01.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:05:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T13:05:01.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:05:01.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T13:05:01.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:05:01.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T13:05:01.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:05:01.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:624: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2024-03-09T13:05:01.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:626: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:05:01.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-03-09T13:05:01.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T13:05:01.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:05:01.690 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T13:05:01.690 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:05:01.691 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:05:01.691 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:05:01.691 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:05:01.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T13:05:01.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:05:01.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T13:05:02.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2024-03-09T13:05:02.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:05:02.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:05:02.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:05:02.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:05:02.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2024-03-09T13:05:02.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:05:02.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:05:02.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:05:02.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:05:02.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:05:02.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:05:02.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:05:02.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:05:02.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:05:02.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:05:02.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:05:02.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2024-03-09T13:05:02.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:05:02.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:05:02.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:05:02.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:05:02.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:05:02.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2024-03-09T13:05:03.773 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-09T13:05:04.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:05:04.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:05:04.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:05:04.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:05:04.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:05:04.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:05:04.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:05:04.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:05:04.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:05:04.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:05:04.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:05:04.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:05:04.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:05:04.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:05:04.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:05:04.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:05:04.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:05:04.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:05:04.299 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:05:04.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:05:04.300 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:05:04.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:05:04.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:05:04.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:05:04.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:05:04.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:05:04.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:05:04.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:05:04.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:05:04.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:05:04.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:05:04.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:05:04.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:05:04.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:05:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:05:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:05:04.302 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:05:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:05:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:05:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:05:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:05:04.325 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:04.328+0000 7fa90fbbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:05:04.327 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:04.330+0000 7fa90fbbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:05:04.328 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:04.332+0000 7fa90fbbd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:05:04.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:05:04.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:05:04.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:05:04.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:05:04.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:05:04.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:04.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:05:04.579 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:05:04.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:04.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:05:04.842 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:04.846+0000 7fa90fbbd640 -1 Falling back to public interface 2024-03-09T13:05:04.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:05:05.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:05:05.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:05.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:05:05.852 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:05:05.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:05.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:05:06.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:05:07.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:05:07.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:07.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:05:07.140 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:05:07.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:07.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:05:07.179 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:07.182+0000 7fa90fbbd640 -1 osd.0 20 log_to_monitors true 2024-03-09T13:05:07.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:05:08.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:05:08.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:08.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:05:08.451 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:05:08.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:08.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:05:08.728 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/4242612485,v1:127.0.0.1:6803/4242612485] [v2:127.0.0.1:6804/4242612485,v1:127.0.0.1:6805/4242612485] exists,up 468df717-fddc-44ba-a73d-6658db3087b5 2024-03-09T13:05:08.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:05:08.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:05:08.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:05:08.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:05:08.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:05:08.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:05:08.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:05:08.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:05:08.728 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:05:08.728 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:05:08.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:05:08.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:05:08.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:05:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:05:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:05:08.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:05:08.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:05:08.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:05:08.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:05:09.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:05:09.086 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:05:09.086 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:05:09.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:05:09.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:09.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:05:09.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-03-09T13:05:09.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-03-09T13:05:09.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-03-09T13:05:09.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:09.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:05:09.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672979 2024-03-09T13:05:09.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672979 2024-03-09T13:05:09.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672979' 2024-03-09T13:05:09.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:09.295 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:05:09.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-03-09T13:05:09.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-03-09T13:05:09.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672979 2-64424509449' 2024-03-09T13:05:09.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:09.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-03-09T13:05:09.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:09.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:05:09.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-03-09T13:05:09.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:09.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-03-09T13:05:09.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-03-09T13:05:09.401 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 103079215106 2024-03-09T13:05:09.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:05:09.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-03-09T13:05:09.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:05:10.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:05:10.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:05:10.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-03-09T13:05:10.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:10.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672979 2024-03-09T13:05:10.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:10.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:05:10.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672979 2024-03-09T13:05:10.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:10.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672979 2024-03-09T13:05:10.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-03-09T13:05:10.958 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672979 2024-03-09T13:05:10.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:05:11.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672979 2024-03-09T13:05:11.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:11.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-03-09T13:05:11.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:11.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:05:11.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-03-09T13:05:11.234 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:11.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-03-09T13:05:11.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-03-09T13:05:11.235 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509449 2024-03-09T13:05:11.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:05:11.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509449 -lt 64424509449 2024-03-09T13:05:11.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:05:11.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:05:11.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:05:11.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:05:11.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:05:11.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:05:11.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:05:11.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:05:11.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:05:11.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:05:11.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:05:12.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:05:12.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:05:12.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:05:12.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:05:12.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:05:12.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:05:12.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:05:12.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-03-09T13:05:12.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:05:12.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:05:12.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:05:12.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:05:12.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-09T13:05:12.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:05:12.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:05:12.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:05:12.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:05:12.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-09T13:05:12.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:05:12.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:05:12.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:05:12.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:05:12.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:05:12.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:05:12.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:05:12.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:05:12.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:05:12.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:05:13.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:05:13.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2024-03-09T13:05:13.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:05:13.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:05:13.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:05:13.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:05:13.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:05:13.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2024-03-09T13:05:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:05:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:05:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:05:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:05:14.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:05:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:05:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:05:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:05:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:05:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:05:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:05:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:05:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:05:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:05:14.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:05:14.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:05:14.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:05:14.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:05:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:05:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:05:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:05:14.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:05:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:05:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:05:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:05:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:05:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:05:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:05:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:05:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:05:14.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:05:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:05:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:05:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:05:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:05:14.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:05:14.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:05:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:05:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:05:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:05:14.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:05:14.893 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:14.894+0000 7f11b1de1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:05:14.894 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:14.897+0000 7f11b1de1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:05:14.895 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:14.898+0000 7f11b1de1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:05:15.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:05:15.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:05:15.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:05:15.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:05:15.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:05:15.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:15.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:05:15.149 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:05:15.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:15.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:05:15.415 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:15.417+0000 7f11b1de1640 -1 Falling back to public interface 2024-03-09T13:05:15.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:05:16.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:05:16.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:16.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:05:16.419 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:05:16.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:16.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:05:16.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:05:17.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:05:17.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:17.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:05:17.697 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:05:17.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:17.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:05:17.739 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:17.741+0000 7f11b1de1640 -1 osd.1 25 log_to_monitors true 2024-03-09T13:05:18.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:05:19.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:05:19.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:19.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:05:19.036 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:05:19.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:19.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:05:19.312 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [10,25) [v2:127.0.0.1:6810/446938368,v1:127.0.0.1:6811/446938368] [v2:127.0.0.1:6812/446938368,v1:127.0.0.1:6813/446938368] exists,up 1b03a655-6f51-4fb8-9a0c-9f1b94afc80e 2024-03-09T13:05:19.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:05:19.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:05:19.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:05:19.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:05:19.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:05:19.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:05:19.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:05:19.312 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:05:19.313 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:05:19.313 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:05:19.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:05:19.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:05:19.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:05:19.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:05:19.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:05:19.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:05:19.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:05:19.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:05:19.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:05:19.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:05:19.669 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:05:19.669 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:05:19.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:05:19.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:19.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:05:19.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215109 2024-03-09T13:05:19.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215109 2024-03-09T13:05:19.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109' 2024-03-09T13:05:19.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:19.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:05:19.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-03-09T13:05:19.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-03-09T13:05:19.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109 1-124554051586' 2024-03-09T13:05:19.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:19.876 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:05:19.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-09T13:05:19.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-09T13:05:19.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215109 1-124554051586 2-64424509452' 2024-03-09T13:05:19.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:19.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215109 2024-03-09T13:05:19.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:19.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:05:19.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215109 2024-03-09T13:05:19.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:19.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215109 2024-03-09T13:05:19.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2024-03-09T13:05:19.984 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 103079215109 2024-03-09T13:05:19.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:05:20.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215109 2024-03-09T13:05:20.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:20.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051586 2024-03-09T13:05:20.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:20.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:05:20.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051586 2024-03-09T13:05:20.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:20.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-03-09T13:05:20.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2024-03-09T13:05:20.260 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 124554051586 2024-03-09T13:05:20.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:05:20.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-03-09T13:05:20.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:05:21.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:05:21.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:05:21.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-03-09T13:05:21.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:05:22.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:05:22.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:05:23.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-03-09T13:05:23.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:23.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-09T13:05:23.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:23.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:05:23.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-09T13:05:23.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:23.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-09T13:05:23.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-09T13:05:23.108 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509452 2024-03-09T13:05:23.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:05:23.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-03-09T13:05:23.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:05:23.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:05:23.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:05:23.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:05:23.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:05:23.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:05:23.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:05:23.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:05:23.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:05:23.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:05:23.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:05:23.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:05:23.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:05:23.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:05:23.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:05:24.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:05:24.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:05:24.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:05:24.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2024-03-09T13:05:24.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T13:05:24.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:05:24.333 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T13:05:24.333 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:05:24.333 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:05:24.334 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:05:24.334 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:05:24.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T13:05:24.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:05:24.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T13:05:24.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:635: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2024-03-09T13:05:24.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:05:24.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:05:24.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:05:24.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:05:24.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2024-03-09T13:05:24.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:05:24.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:05:24.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:05:24.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:05:24.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:05:24.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:05:24.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:05:24.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:05:24.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:05:24.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:05:25.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:05:25.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-09T13:05:25.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:05:25.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:05:25.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:05:25.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:05:25.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:05:25.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-09T13:05:26.544 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:104778fc:::obj2:head# 2024-03-09T13:05:27.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:05:27.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:05:27.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:05:27.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:05:27.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:05:27.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:05:27.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:05:27.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:05:27.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:05:27.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:05:27.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:05:27.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:05:27.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:05:27.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:05:27.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:05:27.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:05:27.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:05:27.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:05:27.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:05:27.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:05:27.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:05:27.065 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:05:27.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:05:27.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:05:27.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:05:27.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:05:27.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:05:27.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:05:27.095 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:27.096+0000 7f931ca20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:05:27.096 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:27.099+0000 7f931ca20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:05:27.097 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:27.100+0000 7f931ca20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:05:27.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:05:27.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:05:27.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:05:27.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:05:27.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:05:27.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:27.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:05:27.350 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:05:27.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:27.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:05:27.616 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:27.619+0000 7f931ca20640 -1 Falling back to public interface 2024-03-09T13:05:27.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:05:28.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:05:28.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:28.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:05:28.620 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:05:28.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:28.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:05:28.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:05:29.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:05:29.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:29.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:05:29.896 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:05:29.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:29.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:05:29.933 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:29.935+0000 7f931ca20640 -1 osd.0 30 log_to_monitors true 2024-03-09T13:05:30.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:05:31.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:05:31.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:31.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:05:31.239 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:05:31.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:31.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:05:31.520 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 34 up_thru 26 down_at 31 last_clean_interval [24,30) [v2:127.0.0.1:6802/202351625,v1:127.0.0.1:6803/202351625] [v2:127.0.0.1:6804/202351625,v1:127.0.0.1:6805/202351625] exists,up 468df717-fddc-44ba-a73d-6658db3087b5 2024-03-09T13:05:31.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:05:31.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:05:31.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:05:31.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:05:31.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:05:31.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:05:31.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:05:31.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:05:31.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:05:31.521 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:05:31.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:05:31.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:05:31.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:05:31.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:05:31.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:05:31.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:05:31.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:05:31.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:05:31.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:05:31.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:05:31.874 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:05:31.874 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:05:31.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:05:31.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:31.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:05:31.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-03-09T13:05:31.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-03-09T13:05:31.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066' 2024-03-09T13:05:31.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:31.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:05:32.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-03-09T13:05:32.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-03-09T13:05:32.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-124554051589' 2024-03-09T13:05:32.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:32.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:05:32.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-09T13:05:32.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-09T13:05:32.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-124554051589 2-64424509455' 2024-03-09T13:05:32.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:32.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888066 2024-03-09T13:05:32.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:32.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:05:32.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888066 2024-03-09T13:05:32.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:32.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-03-09T13:05:32.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-03-09T13:05:32.187 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 146028888066 2024-03-09T13:05:32.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:05:32.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 146028888066 2024-03-09T13:05:32.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:05:33.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:05:33.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:05:33.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 146028888066 2024-03-09T13:05:33.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:05:34.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:05:34.746 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:05:35.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888066 2024-03-09T13:05:35.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:35.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-124554051589 2024-03-09T13:05:35.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:35.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:05:35.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-124554051589 2024-03-09T13:05:35.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:35.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-03-09T13:05:35.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 124554051589' 2024-03-09T13:05:35.028 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 124554051589 2024-03-09T13:05:35.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:05:35.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051590 -lt 124554051589 2024-03-09T13:05:35.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:35.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-09T13:05:35.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:35.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:05:35.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-09T13:05:35.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:35.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-09T13:05:35.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-09T13:05:35.310 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509455 2024-03-09T13:05:35.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:05:35.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-09T13:05:35.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:05:35.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:05:35.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:05:35.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:05:35.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:05:35.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:05:35.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:05:35.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:05:35.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:05:35.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:05:35.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:05:36.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:05:36.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:05:36.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:05:36.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:05:36.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:05:36.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:05:36.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:05:36.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2024-03-09T13:05:36.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:05:36.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:05:36.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:05:36.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:05:36.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-09T13:05:36.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:05:36.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:05:36.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:05:36.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:05:36.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-09T13:05:36.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:05:36.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:05:36.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:05:36.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:05:36.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:05:36.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:05:36.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:05:36.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:05:36.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:05:36.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:05:37.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:05:37.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2024-03-09T13:05:37.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:05:37.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:05:37.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:05:37.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:05:37.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:05:37.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2024-03-09T13:05:38.413 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2024-03-09T13:05:38.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:05:38.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:05:38.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:05:38.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:05:38.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:05:38.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:05:38.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:05:38.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:05:38.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:05:38.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:05:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:05:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:05:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:05:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:05:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:05:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:05:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:05:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:05:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:05:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:05:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:05:38.936 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:05:38.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:05:38.937 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:05:38.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:05:38.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:05:38.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:05:38.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:05:38.965 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:38.969+0000 7f781f215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:05:38.967 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:38.970+0000 7f781f215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:05:38.969 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:38.971+0000 7f781f215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:05:39.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:05:39.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:05:39.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:05:39.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:05:39.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:05:39.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:39.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:05:39.219 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:05:39.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:39.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:05:39.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:39.487+0000 7f781f215640 -1 Falling back to public interface 2024-03-09T13:05:39.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:05:40.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:05:40.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:40.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:05:40.493 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:05:40.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:40.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:05:40.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:05:41.769 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:05:41.771+0000 7f781f215640 -1 osd.1 35 log_to_monitors true 2024-03-09T13:05:41.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:05:41.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:41.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:05:41.777 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:05:41.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:41.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:05:42.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:05:43.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:05:43.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:05:43.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:05:43.057 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:05:43.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:05:43.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:05:43.342 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [29,35) [v2:127.0.0.1:6810/1226716565,v1:127.0.0.1:6811/1226716565] [v2:127.0.0.1:6812/1226716565,v1:127.0.0.1:6813/1226716565] exists,up 1b03a655-6f51-4fb8-9a0c-9f1b94afc80e 2024-03-09T13:05:43.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:05:43.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:05:43.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:05:43.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:05:43.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:05:43.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:05:43.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:05:43.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:05:43.343 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:05:43.343 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:05:43.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:05:43.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:05:43.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:05:43.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:05:43.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:05:43.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:05:43.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:05:43.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:05:43.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:05:43.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:05:43.695 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:05:43.695 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:05:43.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:05:43.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:43.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:05:43.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888069 2024-03-09T13:05:43.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888069 2024-03-09T13:05:43.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888069' 2024-03-09T13:05:43.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:43.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:05:43.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724546 2024-03-09T13:05:43.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724546 2024-03-09T13:05:43.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888069 1-167503724546' 2024-03-09T13:05:43.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:43.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:05:44.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-03-09T13:05:44.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-03-09T13:05:44.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888069 1-167503724546 2-64424509458' 2024-03-09T13:05:44.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:44.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888069 2024-03-09T13:05:44.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:44.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:05:44.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888069 2024-03-09T13:05:44.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:44.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888069 2024-03-09T13:05:44.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888069' 2024-03-09T13:05:44.011 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 146028888069 2024-03-09T13:05:44.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:05:44.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888069 -lt 146028888069 2024-03-09T13:05:44.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:44.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724546 2024-03-09T13:05:44.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:44.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:05:44.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724546 2024-03-09T13:05:44.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:44.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724546 2024-03-09T13:05:44.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2024-03-09T13:05:44.282 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 167503724546 2024-03-09T13:05:44.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:05:44.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 167503724546 2024-03-09T13:05:44.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:05:45.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:05:45.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:05:45.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 167503724546 2024-03-09T13:05:45.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:05:46.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:05:46.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:05:47.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724546 -lt 167503724546 2024-03-09T13:05:47.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:47.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-03-09T13:05:47.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:47.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:05:47.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-03-09T13:05:47.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:47.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-03-09T13:05:47.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-03-09T13:05:47.124 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509458 2024-03-09T13:05:47.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:05:47.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509458 2024-03-09T13:05:47.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:05:47.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:05:47.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:05:47.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:05:47.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:05:47.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:05:47.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:05:47.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:05:47.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:05:47.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:05:47.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:05:48.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:05:48.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:05:48.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:05:48.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:05:48.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:05:48.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:05:48.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:05:48.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2024-03-09T13:05:48.507 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0: { 2024-03-09T13:05:48.508 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-09T13:05:48.508 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T13:05:48.521 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1: { 2024-03-09T13:05:48.522 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2024-03-09T13:05:48.522 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T13:05:48.535 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2: { 2024-03-09T13:05:48.535 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2024-03-09T13:05:48.535 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T13:05:48.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2024-03-09T13:05:48.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T13:05:48.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-09T13:05:48.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-09T13:05:48.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:05:48.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2024-03-09T13:05:48.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2024-03-09T13:05:48.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:05:48.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T13:05:48.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T13:05:48.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:05:49.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2024-03-09T13:05:49.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2024-03-09T13:05:49.140 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:05:49.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:05:49.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:05:49.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:05:49.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2024-03-09T13:04:54.973271+0000 2024-03-09T13:05:49.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2024-03-09T13:05:49.506 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T13:05:49.506 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": true, 2024-03-09T13:05:49.506 INFO:tasks.workunit.client.0.smithi162.stdout: "must": false, 2024-03-09T13:05:49.507 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "2024-02-24T13:04:09.510452+0000" 2024-03-09T13:05:49.507 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T13:05:49.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:645: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2024-03-09T13:04:54.973271+0000 2024-03-09T13:05:49.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T13:05:49.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:04:54.973271+0000 2024-03-09T13:05:49.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:05:49.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:05:49.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:05:49.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:05:49.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:05:49.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:05:49.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:05:49.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:05:49.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:04:54.973271+0000 '>' 2024-03-09T13:04:54.973271+0000 2024-03-09T13:05:49.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:05:50.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:05:50.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:05:50.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:05:50.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:05:50.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:05:50.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:05:50.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:05:51.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:04:54.973271+0000 '>' 2024-03-09T13:04:54.973271+0000 2024-03-09T13:05:51.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:05:52.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:05:52.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:05:52.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:05:52.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:05:52.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:05:52.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:05:52.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:05:52.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:04:54.973271+0000 '>' 2024-03-09T13:04:54.973271+0000 2024-03-09T13:05:52.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:05:53.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:05:53.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:05:53.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:05:53.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:05:53.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:05:53.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:05:53.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:05:53.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:04:54.973271+0000 '>' 2024-03-09T13:04:54.973271+0000 2024-03-09T13:05:53.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:05:54.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:05:54.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:05:54.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:05:54.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:05:54.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:05:54.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:05:54.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:05:54.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:05:50.757559+0000 '>' 2024-03-09T13:04:54.973271+0000 2024-03-09T13:05:54.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:05:54.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2024-03-09T13:05:54.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:05:54.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:05:54.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:05:54.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:05:54.917 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:05:54.917 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:05:54.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:05:54.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:05:54.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:05:55.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:05:55.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:05:55.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:05:55.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:05:55.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:05:55.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:05:55.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:05:55.271 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:05:55.271 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:05:55.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:05:55.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:55.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:05:55.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888073 2024-03-09T13:05:55.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888073 2024-03-09T13:05:55.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888073' 2024-03-09T13:05:55.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:55.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:05:55.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724549 2024-03-09T13:05:55.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724549 2024-03-09T13:05:55.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888073 1-167503724549' 2024-03-09T13:05:55.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:55.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:05:55.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-03-09T13:05:55.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-03-09T13:05:55.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888073 1-167503724549 2-64424509462' 2024-03-09T13:05:55.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:55.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888073 2024-03-09T13:05:55.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:55.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:05:55.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888073 2024-03-09T13:05:55.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:55.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888073 2024-03-09T13:05:55.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888073' 2024-03-09T13:05:55.585 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 146028888073 2024-03-09T13:05:55.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:05:55.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888071 -lt 146028888073 2024-03-09T13:05:55.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:05:56.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:05:56.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:05:57.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888073 -lt 146028888073 2024-03-09T13:05:57.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:57.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724549 2024-03-09T13:05:57.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:57.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:05:57.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724549 2024-03-09T13:05:57.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:57.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724549 2024-03-09T13:05:57.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724549' 2024-03-09T13:05:57.145 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 167503724549 2024-03-09T13:05:57.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:05:57.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724549 -lt 167503724549 2024-03-09T13:05:57.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:57.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-03-09T13:05:57.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:57.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:05:57.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-03-09T13:05:57.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:57.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-03-09T13:05:57.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-03-09T13:05:57.420 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509462 2024-03-09T13:05:57.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:05:57.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-03-09T13:05:57.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:05:57.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:05:57.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:05:58.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:05:58.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:05:58.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:05:58.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:05:58.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:05:58.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:05:58.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:05:58.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:05:58.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:05:58.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:05:58.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:05:58.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:05:58.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:05:58.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:05:58.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:05:58.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2024-03-09T13:05:58.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:05:58.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:05:58.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:05:58.935 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:05:58.935 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:05:58.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:05:58.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:58.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:05:59.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888074 2024-03-09T13:05:59.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888074 2024-03-09T13:05:59.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888074' 2024-03-09T13:05:59.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:59.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:05:59.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724551 2024-03-09T13:05:59.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724551 2024-03-09T13:05:59.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888074 1-167503724551' 2024-03-09T13:05:59.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:05:59.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:05:59.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-03-09T13:05:59.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-03-09T13:05:59.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888074 1-167503724551 2-64424509464' 2024-03-09T13:05:59.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:05:59.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888074 2024-03-09T13:05:59.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:05:59.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:05:59.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888074 2024-03-09T13:05:59.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:05:59.265 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 146028888074 2024-03-09T13:05:59.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888074 2024-03-09T13:05:59.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888074' 2024-03-09T13:05:59.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:05:59.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888073 -lt 146028888074 2024-03-09T13:05:59.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:06:00.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:06:00.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:06:00.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888074 -lt 146028888074 2024-03-09T13:06:00.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:06:00.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-167503724551 2024-03-09T13:06:00.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:06:00.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:06:00.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-167503724551 2024-03-09T13:06:00.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:06:00.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724551 2024-03-09T13:06:00.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 167503724551' 2024-03-09T13:06:00.820 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 167503724551 2024-03-09T13:06:00.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:06:01.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724551 -lt 167503724551 2024-03-09T13:06:01.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:06:01.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-03-09T13:06:01.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:06:01.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:06:01.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-03-09T13:06:01.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:06:01.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-03-09T13:06:01.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-03-09T13:06:01.098 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509464 2024-03-09T13:06:01.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:06:01.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-03-09T13:06:01.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2024-03-09T13:06:01.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-03-09T13:06:01.670 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T13:06:01.670 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2024-03-09T13:05:50.757700+0000 20'10 40:96 [1,0] 1 [1,0] 1 20'10 2024-03-09T13:05:50.757559+0000 20'10 2024-03-09T13:05:50.757559+0000 0 1 periodic scrub scheduled @ 2024-03-10T13:05:50.757559+0000 10 0 2024-03-09T13:06:01.670 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T13:06:01.670 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T13:06:01.671 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T13:06:01.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2024-03-09T13:06:01.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:650: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2024-03-09T13:06:01.936 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T13:06:01.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:06:01.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:06:01.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:06:01.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:06:01.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:06:01.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:06:01.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:06:01.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:06:01.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:06:02.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:06:02.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:06:02.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:06:02.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:06:02.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:06:02.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:06:02.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:06:02.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:06:02.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:06:02.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:06:02.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:06:02.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:06:02.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:06:02.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:06:02.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:06:02.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:06:02.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:06:02.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:06:02.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:06:02.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:06:02.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:06:02.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:06:02.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:06:02.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:06:02.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:06:02.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:06:02.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:06:02.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:06:02.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:06:02.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:06:02.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:06:02.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:06:02.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:06:02.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:06:02.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:06:02.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:06:02.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:06:02.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:06:02.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:06:02.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:06:02.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:06:02.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:06:02.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:06:02.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:06:02.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:06:02.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:06:02.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:06:02.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:06:02.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:06:02.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:06:02.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:06:02.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:06:02.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:06:02.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:06:02.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:06:02.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:06:02.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:06:02.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:06:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:06:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:06:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:06:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2024-03-09T13:06:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:483: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2024-03-09T13:06:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:484: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2024-03-09T13:06:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:487: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2024-03-09T13:06:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:06:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:06:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:06:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:06:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:06:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T13:06:02.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:06:02.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:06:02.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:06:02.562 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:06:02.562 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:06:02.562 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:06:02.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:06:02.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T13:06:02.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:06:02.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:06:02.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:06:02.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:06:02.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:06:02.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:06:02.765 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:06:02.765 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:06:02.765 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:06:02.765 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:06:02.766 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:06:02.766 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:06:02.766 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:06:02.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:06:02.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:06:02.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:06:02.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:06:02.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:06:02.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:06:02.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:06:02.846 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:06:02.846 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:06:02.846 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:06:02.847 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:06:02.847 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:06:02.847 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:06:02.847 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:06:02.847 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:06:02.847 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:06:02.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:488: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2024-03-09T13:06:02.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:06:02.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:06:02.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:06:02.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:06:02.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:06:02.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:06:03.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:06:03.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:06:03.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:06:03.731 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:06:03.733 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:06:03.734 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:06:03.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:06:03.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:06:03.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:06:03.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-09T13:06:03.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: seq 0 2 2024-03-09T13:06:03.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-09T13:06:03.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T13:06:03.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:06:03.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:06:03.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:06:03.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:06:03.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:06:03.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:06:03.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:06:03.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:06:03.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:06:03.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:06:03.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:06:03.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:06:03.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:06:03.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:06:03.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:06:03.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:06:03.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:06:03.766 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:06:03.766 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:06:03.766 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-09T13:06:03.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:06:03.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:06:03.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a01c995d-973f-4ab7-bf6f-661cf6bbc811 2024-03-09T13:06:03.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a01c995d-973f-4ab7-bf6f-661cf6bbc811' 2024-03-09T13:06:03.768 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 a01c995d-973f-4ab7-bf6f-661cf6bbc811 2024-03-09T13:06:03.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:06:03.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC7XuxlIOHDLhAAjDVCUDE5WXcbwVo40vsWLA== 2024-03-09T13:06:03.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC7XuxlIOHDLhAAjDVCUDE5WXcbwVo40vsWLA=="}' 2024-03-09T13:06:03.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a01c995d-973f-4ab7-bf6f-661cf6bbc811 -i td/osd-scrub-repair/0/new.json 2024-03-09T13:06:03.987 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:06:03.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:06:03.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQC7XuxlIOHDLhAAjDVCUDE5WXcbwVo40vsWLA== --osd-uuid a01c995d-973f-4ab7-bf6f-661cf6bbc811 2024-03-09T13:06:04.030 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:04.032+0000 7f88f0c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:06:04.038 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:04.040+0000 7f88f0c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:06:04.039 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:04.041+0000 7f88f0c14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:06:04.040 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:04.042+0000 7f88f0c14640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:06:04.040 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:04.042+0000 7f88f0c14640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:06:04.040 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:04.042+0000 7f88f0c14640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:06:04.130 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:04.132+0000 7f88f0c14640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:06:09.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:06:09.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:06:09.575 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:06:09.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:06:09.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:06:09.935 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:06:10.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:06:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T13:06:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:06:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:06:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:06:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:09.967+0000 7f4a10b24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:06:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:09.983+0000 7f4a10b24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:06:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:09.984+0000 7f4a10b24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:06:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:06:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:06:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:06:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:06:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:06:10.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:10.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:06:10.237 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:06:10.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:10.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:10.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:10.505 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:10.508+0000 7f4a10b24640 -1 Falling back to public interface 2024-03-09T13:06:11.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:11.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:11.485 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:06:11.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:06:11.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:11.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:11.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:12.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:12.764 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:06:12.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:12.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:06:12.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:12.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:12.771 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:12.774+0000 7f4a10b24640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:06:13.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:14.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:14.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:14.057 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:06:14.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:06:14.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:14.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:14.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:15.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:15.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:15.342 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:06:15.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:06:15.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:15.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:15.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:16.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:16.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:16.675 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:06:16.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:06:16.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:16.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:16.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:17.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:17.955 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:06:17.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:17.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:06:17.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:17.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:18.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:19.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:19.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:19.242 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:06:19.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:06:19.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:19.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:19.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:20.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:20.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:20.602 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:06:20.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:06:20.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:20.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:20.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:21.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:21.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:21.885 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:06:21.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:06:21.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:21.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:22.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:23.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:23.170 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:06:23.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:23.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:06:23.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:23.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:23.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:24.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:24.806 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:06:24.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:24.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:06:24.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:24.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:25.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:26.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:26.371 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:06:26.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:26.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:06:26.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:26.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:26.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:27.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:27.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:27.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:06:27.757 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:06:27.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:27.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:28.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:29.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:29.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:29.501 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:06:29.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:06:29.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:29.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:29.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:30.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:30.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:30.909 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:06:30.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:06:30.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:30.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:31.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:32.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:32.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:32.539 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:06:32.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:06:32.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:32.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:33.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:34.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:34.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:34.192 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:06:34.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:06:34.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:34.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:06:34.467 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1005274148,v1:127.0.0.1:6803/1005274148] [v2:127.0.0.1:6804/1005274148,v1:127.0.0.1:6805/1005274148] exists,up a01c995d-973f-4ab7-bf6f-661cf6bbc811 2024-03-09T13:06:34.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:06:34.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:06:34.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:06:34.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-09T13:06:34.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T13:06:34.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:06:34.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:06:34.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:06:34.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:06:34.474 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:06:34.474 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:06:34.474 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:06:34.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:06:34.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:06:34.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:06:34.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:06:34.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:06:34.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:06:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:06:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:06:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:06:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:06:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:06:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:06:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-09T13:06:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:06:34.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:06:34.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=db94a355-5ad0-45b7-9aa7-27aa68b4c6ad 2024-03-09T13:06:34.475 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 db94a355-5ad0-45b7-9aa7-27aa68b4c6ad 2024-03-09T13:06:34.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 db94a355-5ad0-45b7-9aa7-27aa68b4c6ad' 2024-03-09T13:06:34.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:06:34.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDaXuxlVqJeHRAAJpDJ72jJ9dcCIcHUsFHLQg== 2024-03-09T13:06:34.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDaXuxlVqJeHRAAJpDJ72jJ9dcCIcHUsFHLQg=="}' 2024-03-09T13:06:34.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new db94a355-5ad0-45b7-9aa7-27aa68b4c6ad -i td/osd-scrub-repair/1/new.json 2024-03-09T13:06:34.885 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:06:34.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:06:34.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQDaXuxlVqJeHRAAJpDJ72jJ9dcCIcHUsFHLQg== --osd-uuid db94a355-5ad0-45b7-9aa7-27aa68b4c6ad 2024-03-09T13:06:34.928 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:34.931+0000 7f6e9361f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:06:34.969 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:34.973+0000 7f6e9361f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:06:34.970 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:34.974+0000 7f6e9361f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:06:34.971 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:34.974+0000 7f6e9361f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:06:34.971 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:34.974+0000 7f6e9361f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:06:34.971 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:34.974+0000 7f6e9361f640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:06:35.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:35.039+0000 7f6e9361f640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:06:40.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:06:40.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:06:40.851 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:06:40.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:06:40.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:06:41.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:06:41.206 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:06:41.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T13:06:41.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:06:41.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:06:41.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:06:41.234 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:41.237+0000 7fdc18e10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:06:41.235 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:41.239+0000 7fdc18e10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:06:41.236 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:41.240+0000 7fdc18e10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:06:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:06:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:06:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:06:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:06:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:06:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:41.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:06:41.479 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:06:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:41.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:06:41.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:41.764 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:41.768+0000 7fdc18e10640 -1 Falling back to public interface 2024-03-09T13:06:42.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:42.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:42.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:06:42.753 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:06:42.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:42.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:06:43.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:44.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:44.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:44.035 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:06:44.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:06:44.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:44.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:06:44.209 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:06:44.212+0000 7fdc18e10640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:06:44.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:45.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:45.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:45.402 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:06:45.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:06:45.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:45.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:06:45.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:46.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:46.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:46.689 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:06:46.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:06:46.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:46.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:06:47.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:48.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:48.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:48.069 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:06:48.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:06:48.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:48.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:06:48.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:49.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:49.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:49.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:06:49.402 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:06:49.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:49.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:06:49.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:50.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:50.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:50.687 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:06:50.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:06:50.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:50.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:06:51.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:52.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:52.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:52.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:06:52.027 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:06:52.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:52.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:06:52.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:53.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:53.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:53.397 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:06:53.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:06:53.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:53.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:06:53.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:54.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:54.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:54.684 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:06:54.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:06:54.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:54.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:06:55.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:56.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:56.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:56.033 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:06:56.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:06:56.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:56.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:06:56.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:57.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:57.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:57.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:06:57.645 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:06:57.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:57.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:06:58.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:06:59.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:06:59.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:06:59.339 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:06:59.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:06:59.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:06:59.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:07:00.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:01.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:01.008 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:07:01.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:01.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:07:01.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:01.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:07:01.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:02.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:02.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:02.669 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:07:02.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:07:02.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:02.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:07:03.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:04.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:04.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:04.446 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:07:04.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:07:04.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:04.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:07:04.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:05.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:05.969 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:07:05.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:05.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:07:05.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:05.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:07:06.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:07.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:07.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:07.347 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:07:07.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:07:07.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:07.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:07:07.623 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2704321576,v1:127.0.0.1:6811/2704321576] [v2:127.0.0.1:6812/2704321576,v1:127.0.0.1:6813/2704321576] exists,up db94a355-5ad0-45b7-9aa7-27aa68b4c6ad 2024-03-09T13:07:07.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:07:07.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:07:07.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:07:07.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2024-03-09T13:07:07.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:494: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T13:07:07.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:07:07.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:07:07.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T13:07:07.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:07:07.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:07:07.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:07:07.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:07:07.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:07:07.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:07:07.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:07:07.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:07:07.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:07:07.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:07:07.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:07:07.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:07:07.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:07:07.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:07:07.631 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:07:07.631 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:07:07.631 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-09T13:07:07.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:07:07.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:07:07.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a8416da2-3ed7-4f18-81a3-54b352d422e7 2024-03-09T13:07:07.633 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 a8416da2-3ed7-4f18-81a3-54b352d422e7 2024-03-09T13:07:07.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a8416da2-3ed7-4f18-81a3-54b352d422e7' 2024-03-09T13:07:07.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:07:07.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7Xuxl4AeSJhAANuNmmOacLHQWNwP+AUBn/A== 2024-03-09T13:07:07.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7Xuxl4AeSJhAANuNmmOacLHQWNwP+AUBn/A=="}' 2024-03-09T13:07:07.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a8416da2-3ed7-4f18-81a3-54b352d422e7 -i td/osd-scrub-repair/2/new.json 2024-03-09T13:07:08.012 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:07:08.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T13:07:08.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQD7Xuxl4AeSJhAANuNmmOacLHQWNwP+AUBn/A== --osd-uuid a8416da2-3ed7-4f18-81a3-54b352d422e7 2024-03-09T13:07:08.055 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:08.058+0000 7fcfb3793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:07:08.104 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:08.107+0000 7fcfb3793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:07:08.105 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:08.108+0000 7fcfb3793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:07:08.106 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:08.108+0000 7fcfb3793640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:07:08.106 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:08.108+0000 7fcfb3793640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:07:08.106 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:08.108+0000 7fcfb3793640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:07:08.171 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:08.174+0000 7fcfb3793640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T13:07:13.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T13:07:13.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:07:13.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T13:07:13.825 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T13:07:13.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:07:14.199 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T13:07:14.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T13:07:14.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T13:07:14.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:07:14.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:07:14.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:07:14.228 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:14.231+0000 7fe862d1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:07:14.229 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:14.233+0000 7fe862d1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:07:14.230 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:14.234+0000 7fe862d1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:07:14.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T13:07:14.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:07:14.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:07:14.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:07:14.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:07:14.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:14.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:07:14.471 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:07:14.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:14.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:14.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:14.755 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:14.759+0000 7fe862d1f640 -1 Falling back to public interface 2024-03-09T13:07:15.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:15.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:15.747 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:07:15.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:07:15.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:15.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:16.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:17.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:17.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:17.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:07:17.030 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:07:17.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:17.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:17.049 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:17.053+0000 7fe862d1f640 -1 osd.2 0 log_to_monitors true 2024-03-09T13:07:17.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:18.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:18.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:18.364 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:07:18.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:07:18.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:18.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:18.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:19.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:19.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:19.645 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:07:19.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:07:19.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:19.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:19.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:20.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:20.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:20.931 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:07:20.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:07:20.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:20.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:21.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:22.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:22.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:22.217 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:07:22.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:07:22.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:22.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:22.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:23.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:23.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:23.502 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:07:23.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:07:23.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:23.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:23.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:24.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:24.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:24.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:07:24.861 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:07:24.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:24.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:25.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:26.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:26.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:26.146 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:07:26.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:07:26.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:26.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:26.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:27.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:27.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:27.430 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:07:27.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:07:27.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:27.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:27.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:28.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:28.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:28.715 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:07:28.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:07:28.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:28.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:29.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:30.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:30.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:30.321 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:07:30.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:07:30.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:30.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:30.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:31.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:31.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:31.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:07:31.809 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:07:31.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:31.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:32.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:33.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:33.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:33.588 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:07:33.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:07:33.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:33.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:35.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:35.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:35.037 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:07:35.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:07:35.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:35.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:35.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:36.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:36.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:36.604 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:07:36.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:07:36.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:36.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:37.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:38.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:38.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:38.222 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:07:38.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:07:38.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:38.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:38.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:39.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:39.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:39.967 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:07:39.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:07:39.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:39.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:07:40.280 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2160863871,v1:127.0.0.1:6819/2160863871] [v2:127.0.0.1:6820/2160863871,v1:127.0.0.1:6821/2160863871] exists,up a8416da2-3ed7-4f18-81a3-54b352d422e7 2024-03-09T13:07:40.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:07:40.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:07:40.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:07:40.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:497: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2024-03-09T13:07:40.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-09T13:07:40.652 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'testpool' created 2024-03-09T13:07:40.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:07:41.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2024-03-09T13:07:42.444 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 1 size to 2 2024-03-09T13:07:42.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-03-09T13:07:42.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:07:42.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:07:42.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:07:42.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:07:42.458 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:07:42.458 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:07:42.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:07:42.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:07:42.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:07:42.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:07:42.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:07:42.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:07:42.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:07:42.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:07:42.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:07:42.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:07:42.834 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:07:42.834 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:07:42.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:07:42.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:07:42.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:07:42.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-09T13:07:42.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-09T13:07:42.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-09T13:07:42.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:07:42.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:07:43.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672971 2024-03-09T13:07:43.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672971 2024-03-09T13:07:43.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672971' 2024-03-09T13:07:43.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:07:43.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:07:43.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-03-09T13:07:43.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-03-09T13:07:43.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-42949672971 2-64424509446' 2024-03-09T13:07:43.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:07:43.218 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-09T13:07:43.218 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:07:43.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:07:43.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-09T13:07:43.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:07:43.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-09T13:07:43.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-09T13:07:43.220 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836499 2024-03-09T13:07:43.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:07:43.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-03-09T13:07:43.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:07:44.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:07:44.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:07:44.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-03-09T13:07:44.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:07:44.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672971 2024-03-09T13:07:44.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:07:44.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:07:44.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672971 2024-03-09T13:07:44.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:07:44.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672971 2024-03-09T13:07:44.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2024-03-09T13:07:44.838 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672971 2024-03-09T13:07:44.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:07:45.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672971 -lt 42949672971 2024-03-09T13:07:45.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:07:45.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-03-09T13:07:45.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:07:45.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:07:45.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-03-09T13:07:45.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:07:45.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-03-09T13:07:45.114 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509446 2024-03-09T13:07:45.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-03-09T13:07:45.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:07:45.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-03-09T13:07:45.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:07:45.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:07:45.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:07:45.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:07:45.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:07:45.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:07:45.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:07:45.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:07:45.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:07:45.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:07:45.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:07:46.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T13:07:46.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:07:46.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:07:46.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:07:46.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T13:07:46.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-09T13:07:46.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:07:46.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-09T13:07:46.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:07:46.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:07:46.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:07:46.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:07:46.441 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:07:46.441 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:07:46.441 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:07:46.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:07:46.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:07:46.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:07:46.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T13:07:46.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:07:46.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:07:46.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:07:47.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T13:07:47.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-09T13:07:47.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:07:47.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:07:47.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:07:47.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:07:47.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:07:47.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:07:47.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:07:47.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:07:47.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:07:47.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:07:47.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:07:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:07:47.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:07:47.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:07:47.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:07:47.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:07:47.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:07:47.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:07:47.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:07:47.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:07:47.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:07:47.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:07:47.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:07:47.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:07:48.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:07:48.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:07:48.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:07:48.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:502: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2024-03-09T13:07:48.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:503: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2024-03-09T13:07:48.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:504: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T13:07:48.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-03-09T13:07:48.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T13:07:48.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:07:48.329 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T13:07:48.329 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:07:48.329 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:07:48.330 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:07:48.330 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:07:48.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T13:07:48.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:07:48.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T13:07:48.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:508: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T13:07:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:07:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:07:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:07:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:07:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T13:07:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:07:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:07:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:07:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:07:48.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:07:48.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:07:48.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:07:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:07:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:07:48.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:07:49.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:07:49.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T13:07:49.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:07:49.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:07:49.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:07:49.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:07:49.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:07:49.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-09T13:07:50.457 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-09T13:07:50.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T13:07:50.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:07:50.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:07:50.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:07:50.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:07:50.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:07:50.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:07:50.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:07:50.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:07:50.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:07:50.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:07:50.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:07:50.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:07:50.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:07:50.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:07:50.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:07:50.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:07:50.976 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:07:50.976 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:07:50.976 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:07:50.976 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:07:50.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:07:50.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:07:50.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:07:50.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:07:50.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:07:50.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:07:50.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:07:50.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:07:50.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:07:50.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:07:50.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-09T13:07:50.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:07:50.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:07:50.978 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:07:50.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T13:07:50.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:07:50.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:07:50.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:07:50.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:07:50.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:07:51.007 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:51.010+0000 7f7959c04640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:07:51.009 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:51.012+0000 7f7959c04640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:07:51.011 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:51.013+0000 7f7959c04640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:07:51.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:07:51.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:07:51.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:07:51.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:07:51.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:07:51.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:51.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:07:51.259 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:07:51.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:51.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:07:51.530 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:51.533+0000 7f7959c04640 -1 Falling back to public interface 2024-03-09T13:07:51.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:52.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:52.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:52.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:07:52.532 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:07:52.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:52.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:07:52.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:53.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:53.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:07:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:07:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:07:53.920 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:07:53.923+0000 7f7959c04640 -1 osd.0 21 log_to_monitors true 2024-03-09T13:07:54.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:07:55.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:07:55.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:07:55.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:07:55.176 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:07:55.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:07:55.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:07:55.492 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3514227291,v1:127.0.0.1:6803/3514227291] [v2:127.0.0.1:6804/3514227291,v1:127.0.0.1:6805/3514227291] exists,up a01c995d-973f-4ab7-bf6f-661cf6bbc811 2024-03-09T13:07:55.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:07:55.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:07:55.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:07:55.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:07:55.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:07:55.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:07:55.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:07:55.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:07:55.493 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:07:55.493 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:07:55.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:07:55.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:07:55.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:07:55.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:07:55.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:07:55.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:07:55.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:07:55.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:07:55.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:07:55.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:07:55.849 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:07:55.849 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:07:55.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:07:55.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:07:55.849 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:07:55.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182402 2024-03-09T13:07:55.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182402 2024-03-09T13:07:55.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402' 2024-03-09T13:07:55.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:07:55.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:07:56.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-03-09T13:07:56.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-03-09T13:07:56.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672974' 2024-03-09T13:07:56.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:07:56.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:07:56.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-03-09T13:07:56.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-03-09T13:07:56.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182402 1-42949672974 2-64424509450' 2024-03-09T13:07:56.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:07:56.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182402 2024-03-09T13:07:56.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:07:56.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:07:56.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182402 2024-03-09T13:07:56.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:07:56.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182402 2024-03-09T13:07:56.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182402' 2024-03-09T13:07:56.176 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 107374182402 2024-03-09T13:07:56.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:07:56.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-09T13:07:56.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:07:57.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:07:57.451 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:07:57.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 107374182402 2024-03-09T13:07:57.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:07:58.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:07:58.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:07:59.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182402 -lt 107374182402 2024-03-09T13:07:59.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:07:59.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-03-09T13:07:59.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:07:59.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:07:59.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-03-09T13:07:59.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:07:59.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-03-09T13:07:59.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-03-09T13:07:59.018 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672974 2024-03-09T13:07:59.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:07:59.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672974 2024-03-09T13:07:59.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:07:59.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-03-09T13:07:59.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:07:59.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:07:59.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-03-09T13:07:59.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:07:59.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-03-09T13:07:59.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-03-09T13:07:59.295 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509450 2024-03-09T13:07:59.295 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:07:59.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-03-09T13:07:59.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:07:59.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:07:59.571 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:07:59.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:07:59.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:07:59.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:07:59.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:07:59.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:07:59.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:07:59.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:07:59.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:08:00.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:08:00.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:08:00.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:08:00.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:08:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:08:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:08:00.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:08:00.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2024-03-09T13:08:00.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T13:08:00.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:08:00.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-09T13:08:00.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:08:00.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2024-03-09T13:08:00.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2024-03-09T13:08:00.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:08:00.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:08:00.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:08:00.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:08:01.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:511: TEST_auto_repair_bluestore_scrub: local primary=1 2024-03-09T13:08:01.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2024-03-09T13:08:01.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:08:01.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:08:01.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:08:01.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:08:01.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:512: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2024-03-09T13:07:40.601549+0000 2024-03-09T13:08:01.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2024-03-09T13:08:01.546 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T13:08:01.546 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2024-03-09T13:08:01.546 INFO:tasks.workunit.client.0.smithi162.stdout: "must": false, 2024-03-09T13:08:01.547 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "2024-03-02T13:06:21.550596+0000" 2024-03-09T13:08:01.547 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T13:08:01.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:516: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2024-03-09T13:07:40.601549+0000 2024-03-09T13:08:01.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T13:08:01.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:07:40.601549+0000 2024-03-09T13:08:01.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:08:01.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:08:01.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:08:01.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:08:01.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:08:01.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:08:01.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:08:01.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:08:01.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-02T13:06:21.550596+0000 '>' 2024-03-09T13:07:40.601549+0000 2024-03-09T13:08:01.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:08:02.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:08:02.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:08:02.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:08:02.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:08:02.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:08:02.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:08:02.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:08:03.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-02T13:06:21.550596+0000 '>' 2024-03-09T13:07:40.601549+0000 2024-03-09T13:08:03.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:08:04.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:08:04.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:08:04.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:08:04.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:08:04.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:08:04.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:08:04.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:08:04.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-02T13:06:21.550596+0000 '>' 2024-03-09T13:07:40.601549+0000 2024-03-09T13:08:04.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:08:05.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:08:05.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:08:05.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:08:05.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:08:05.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:08:05.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:08:05.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:08:05.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-02T13:06:21.550596+0000 '>' 2024-03-09T13:07:40.601549+0000 2024-03-09T13:08:05.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:08:06.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:08:06.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:08:06.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:08:06.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:08:06.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:08:06.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:08:06.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:08:06.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-02T13:06:21.550596+0000 '>' 2024-03-09T13:07:40.601549+0000 2024-03-09T13:08:06.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:08:07.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:08:07.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:08:07.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T13:08:07.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:08:07.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:08:07.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:08:07.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:08:08.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:08:03.293246+0000 '>' 2024-03-09T13:07:40.601549+0000 2024-03-09T13:08:08.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:08:08.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:517: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-09T13:08:08.493 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T13:08:08.493 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-09T13:08:03.493329+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-03-09T13:08:03.293246+0000 21'1 2024-03-09T13:08:03.293246+0000 0 1 periodic scrub scheduled @ 2024-03-10T13:08:03.293246+0000 1 0 2024-03-09T13:08:08.493 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T13:08:08.493 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T13:08:08.493 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T13:08:08.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_scrub: sleep 2 2024-03-09T13:08:10.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-09T13:08:10.773 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T13:08:10.773 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-09T13:08:03.493329+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-03-09T13:08:03.293246+0000 21'1 2024-03-09T13:08:03.293246+0000 0 1 periodic scrub scheduled @ 2024-03-10T13:08:03.293246+0000 1 0 2024-03-09T13:08:10.773 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T13:08:10.773 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T13:08:10.773 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T13:08:10.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:521: TEST_auto_repair_bluestore_scrub: sleep 2 2024-03-09T13:08:12.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-09T13:08:13.052 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T13:08:13.053 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-09T13:08:03.493329+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-03-09T13:08:03.293246+0000 21'1 2024-03-09T13:08:03.293246+0000 0 1 periodic scrub scheduled @ 2024-03-10T13:08:03.293246+0000 1 0 2024-03-09T13:08:13.053 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T13:08:13.053 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T13:08:13.053 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T13:08:13.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_scrub: sleep 5 2024-03-09T13:08:18.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_scrub: wait_for_clean 2024-03-09T13:08:18.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:08:18.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:08:18.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:08:18.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:08:18.066 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:08:18.067 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:08:18.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:08:18.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:08:18.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:08:18.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:08:18.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:08:18.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:08:18.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:08:18.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:08:18.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:08:18.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:08:18.438 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:08:18.439 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:08:18.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:08:18.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:08:18.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:08:18.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182407 2024-03-09T13:08:18.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182407 2024-03-09T13:08:18.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407' 2024-03-09T13:08:18.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:08:18.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:08:18.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-09T13:08:18.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-09T13:08:18.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672980' 2024-03-09T13:08:18.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:08:18.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:08:18.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-09T13:08:18.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-09T13:08:18.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-107374182407 1-42949672980 2-64424509455' 2024-03-09T13:08:18.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:08:18.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-107374182407 2024-03-09T13:08:18.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:08:18.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:08:18.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-107374182407 2024-03-09T13:08:18.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:08:18.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182407 2024-03-09T13:08:18.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 107374182407' 2024-03-09T13:08:18.792 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 107374182407 2024-03-09T13:08:18.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:08:19.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182406 -lt 107374182407 2024-03-09T13:08:19.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:08:20.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:08:20.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:08:20.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182408 -lt 107374182407 2024-03-09T13:08:20.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:08:20.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-09T13:08:20.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:08:20.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:08:20.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-09T13:08:20.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:08:20.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-09T13:08:20.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-09T13:08:20.354 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672980 2024-03-09T13:08:20.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:08:20.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-03-09T13:08:20.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:08:20.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-09T13:08:20.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:08:20.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:08:20.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-09T13:08:20.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:08:20.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-09T13:08:20.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-09T13:08:20.629 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509455 2024-03-09T13:08:20.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:08:20.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-09T13:08:20.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:08:20.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:08:20.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:08:21.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:08:21.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:08:21.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:08:21.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:08:21.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:08:21.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:08:21.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:08:21.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:08:21.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:08:21.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:08:21.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:08:21.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:08:21.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:08:21.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:08:21.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:08:21.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:525: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2024-03-09T13:08:22.139 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T13:08:22.139 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2024-03-09T13:08:03.493329+0000 21'1 26:64 [1,0] 1 [1,0] 1 21'1 2024-03-09T13:08:03.293246+0000 21'1 2024-03-09T13:08:03.293246+0000 0 1 periodic scrub scheduled @ 2024-03-10T13:08:03.293246+0000 1 0 2024-03-09T13:08:22.139 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T13:08:22.139 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T13:08:22.139 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T13:08:22.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2024-03-09T13:08:22.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T13:08:22.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:08:22.152 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T13:08:22.152 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:08:22.152 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:08:22.152 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:08:22.152 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:08:22.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T13:08:22.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:08:22.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T13:08:22.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:528: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T13:08:22.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:08:22.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:08:22.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:08:22.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:08:22.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T13:08:22.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:08:22.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:08:22.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:08:22.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:08:22.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:08:22.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:08:22.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:08:22.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:08:22.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:08:22.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:08:23.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:08:23.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T13:08:23.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:08:23.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:08:23.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:08:23.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:08:23.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:08:23.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-09T13:08:24.235 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:08:24.235 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:08:24.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T13:08:24.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:08:24.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:08:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:08:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:08:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:08:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:08:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:08:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:08:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:08:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:08:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:08:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:08:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:08:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:08:24.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:08:24.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:08:24.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2024-03-09T13:08:24.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:08:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:08:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:08:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2024-03-09T13:08:24.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:08:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:08:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:08:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:08:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:08:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:24.542+0000 7f546df47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:24.542 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:24.545+0000 7f546df47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:24.543 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:24.546+0000 7f546df47640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:24.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:08:24.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:08:24.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:08:24.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:08:24.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:08:24.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:08:24.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:08:24.794 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:08:24.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:08:24.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:08:25.062 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:25.065+0000 7f546df47640 -1 Falling back to public interface 2024-03-09T13:08:25.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:08:26.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:08:26.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:08:26.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:08:26.110 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:08:26.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:08:26.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:08:26.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:08:27.314 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:27.317+0000 7f546df47640 -1 osd.0 26 log_to_monitors true 2024-03-09T13:08:27.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:08:27.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:08:27.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:08:27.388 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:08:27.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:08:27.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:08:27.753 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [25,26) [v2:127.0.0.1:6802/3832803642,v1:127.0.0.1:6803/3832803642] [v2:127.0.0.1:6804/3832803642,v1:127.0.0.1:6805/3832803642] exists,up a01c995d-973f-4ab7-bf6f-661cf6bbc811 2024-03-09T13:08:27.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:08:27.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:08:27.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:08:27.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:08:27.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:08:27.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:08:27.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:08:27.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:08:27.754 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:08:27.754 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:08:27.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:08:27.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:08:27.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:08:27.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:08:27.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:08:27.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:08:27.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:08:27.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:08:27.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:08:28.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:08:28.124 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:08:28.124 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:08:28.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:08:28.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:08:28.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:08:28.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018882 2024-03-09T13:08:28.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018882 2024-03-09T13:08:28.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882' 2024-03-09T13:08:28.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:08:28.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:08:28.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672983 2024-03-09T13:08:28.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672983 2024-03-09T13:08:28.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672983' 2024-03-09T13:08:28.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:08:28.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:08:28.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-03-09T13:08:28.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-03-09T13:08:28.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-128849018882 1-42949672983 2-64424509458' 2024-03-09T13:08:28.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:08:28.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-128849018882 2024-03-09T13:08:28.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:08:28.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:08:28.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-128849018882 2024-03-09T13:08:28.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:08:28.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018882 2024-03-09T13:08:28.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2024-03-09T13:08:28.445 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 128849018882 2024-03-09T13:08:28.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:08:28.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 128849018882 2024-03-09T13:08:28.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:08:29.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:08:29.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:08:30.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018882 -lt 128849018882 2024-03-09T13:08:30.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:08:30.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672983 2024-03-09T13:08:30.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:08:30.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:08:30.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672983 2024-03-09T13:08:30.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:08:30.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672983 2024-03-09T13:08:30.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-03-09T13:08:30.097 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672983 2024-03-09T13:08:30.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:08:30.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672983 -lt 42949672983 2024-03-09T13:08:30.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:08:30.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-03-09T13:08:30.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:08:30.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:08:30.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-03-09T13:08:30.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:08:30.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-03-09T13:08:30.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-03-09T13:08:30.375 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509458 2024-03-09T13:08:30.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:08:30.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-03-09T13:08:30.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:08:30.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:08:30.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:08:30.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:08:30.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:08:30.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:08:30.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:08:30.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:08:30.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:08:30.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:08:30.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:08:31.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T13:08:31.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:08:31.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:08:31.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:08:31.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T13:08:31.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-09T13:08:31.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:08:31.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-09T13:08:31.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:08:31.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:08:31.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:08:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:08:31.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:08:31.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:08:31.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:08:31.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:08:31.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:08:31.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:08:31.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:08:31.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:08:31.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:08:31.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:08:32.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:08:32.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:08:32.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:08:32.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:529: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:08:32.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:08:32.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-09T13:08:32.360 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:08:02.264+0000 7fdbfc6a9640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] planned DEEP_SCRUB_ON_ERROR] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2024-03-09T13:08:32.360 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:08:02.264+0000 7fdbfc6a9640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: DEEP_SCRUB_ON_ERROR ] ] scrubber: scrub_finish Try to auto repair after scrub errors 2024-03-09T13:08:32.360 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:08:02.264+0000 7fdbfc6a9640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing [ 1.0: ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2024-03-09T13:08:32.360 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:08:03.292+0000 7fdbfc6a9640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T13:08:32.360 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:08:03.292+0000 7fdbfc6a9640 10 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T13:08:32.360 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:08:03.292+0000 7fdbfc6a9640 15 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-09T13:08:32.360 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:08:03.292+0000 7fdbfc6a9640 20 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-09T13:08:32.360 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:08:03.292+0000 7fdbfc6a9640 19 osd.1 pg_epoch: 26 pg[1.0( v 21'1 (0'0,21'1] local-lis/les=25/26 n=1 ec=17/17 lis/c=25/25 les/c/f=26/26/0 sis=25) [1,0] r=0 lpr=25 crt=21'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T13:08:32.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2024-03-09T13:08:32.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2024-03-09T13:08:32.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:534: TEST_auto_repair_bluestore_scrub: COUNT=1 2024-03-09T13:08:32.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_scrub: test 1 = 1 2024-03-09T13:08:32.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:08:32.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:08:32.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:08:32.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:08:32.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:08:32.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:08:32.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:08:32.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:08:32.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:08:32.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:08:32.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:08:32.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:08:32.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:08:32.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:08:32.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:08:32.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:08:32.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:08:32.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:08:32.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:08:32.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:08:32.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:08:32.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:08:32.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:08:32.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:08:32.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:08:32.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:08:32.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:08:32.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:08:32.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:08:32.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:08:32.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:08:32.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:08:32.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:08:32.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:08:32.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:08:32.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:08:32.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:08:32.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:08:32.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:08:32.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:08:32.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:08:32.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:08:32.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:08:32.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:08:32.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:08:32.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:08:32.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:08:32.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:08:32.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:08:32.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:08:32.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:08:32.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:08:32.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:08:32.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:08:32.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:08:32.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:08:32.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:08:32.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:08:32.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:08:32.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:08:32.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:08:32.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:08:32.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:08:32.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:08:32.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:08:32.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:08:32.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:08:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:08:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:08:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:08:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2024-03-09T13:08:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2024-03-09T13:08:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: TEST_auto_repair_bluestore_tag: local poolname=testpool 2024-03-09T13:08:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2024-03-09T13:08:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:08:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:08:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:08:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:08:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:08:32.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T13:08:33.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:08:33.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:08:33.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:08:33.068 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:08:33.068 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:08:33.068 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:08:33.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:08:33.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T13:08:33.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:08:33.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:08:33.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:08:33.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:08:33.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:08:33.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:08:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:08:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:08:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:08:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:08:33.253 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:08:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:08:33.254 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:08:33.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:08:33.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:08:33.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:08:33.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:08:33.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:08:33.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:08:33.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:08:33.335 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:08:33.335 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:08:33.335 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:08:33.335 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:08:33.335 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:08:33.335 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:08:33.335 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:08:33.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:08:33.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:08:33.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:388: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2024-03-09T13:08:33.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:08:33.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:08:33.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:08:33.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:08:33.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:08:33.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:08:34.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:08:34.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:08:34.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:08:34.221 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:08:34.221 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:08:34.221 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:08:34.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:08:34.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:08:34.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:08:34.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:395: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T13:08:34.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: seq 0 2 2024-03-09T13:08:34.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-09T13:08:34.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T13:08:34.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:08:34.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:08:34.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:08:34.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:08:34.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:08:34.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:08:34.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:08:34.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:08:34.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:08:34.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:08:34.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:08:34.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:08:34.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:08:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:08:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:08:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:08:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:08:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:08:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T13:08:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:08:34.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:08:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c759759c-608d-48cc-b81b-07672690e9f7 2024-03-09T13:08:34.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c759759c-608d-48cc-b81b-07672690e9f7' 2024-03-09T13:08:34.256 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 c759759c-608d-48cc-b81b-07672690e9f7 2024-03-09T13:08:34.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:08:34.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSX+xlhdtfEBAA2lgVO42+OmbtJlgLhmKP1w== 2024-03-09T13:08:34.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSX+xlhdtfEBAA2lgVO42+OmbtJlgLhmKP1w=="}' 2024-03-09T13:08:34.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c759759c-608d-48cc-b81b-07672690e9f7 -i td/osd-scrub-repair/0/new.json 2024-03-09T13:08:34.460 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:08:34.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:08:34.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBSX+xlhdtfEBAA2lgVO42+OmbtJlgLhmKP1w== --osd-uuid c759759c-608d-48cc-b81b-07672690e9f7 2024-03-09T13:08:34.501 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:34.504+0000 7f4aa1cf7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:34.503 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:34.506+0000 7f4aa1cf7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:34.504 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:34.507+0000 7f4aa1cf7640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:34.505 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:34.508+0000 7f4aa1cf7640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:08:34.505 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:34.508+0000 7f4aa1cf7640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:08:34.505 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:34.508+0000 7f4aa1cf7640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:08:34.569 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:34.571+0000 7f4aa1cf7640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:08:40.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:08:40.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:08:40.117 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:08:40.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:08:40.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:08:40.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:08:40.479 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:08:40.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T13:08:40.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:08:40.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:08:40.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:08:40.508 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:40.511+0000 7f29da7a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:40.510 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:40.513+0000 7f29da7a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:40.511 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:40.514+0000 7f29da7a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:40.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:08:40.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:08:40.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:08:40.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:08:40.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:08:40.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:08:40.753 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:08:40.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:08:40.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:08:40.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:08:41.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:08:41.507 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:41.035+0000 7f29da7a8640 -1 Falling back to public interface 2024-03-09T13:08:42.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:08:42.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:08:42.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:08:42.025 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:08:42.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:08:42.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:08:42.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:08:43.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:08:43.297 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:08:43.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:08:43.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:08:43.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:08:43.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:08:43.449 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:43.452+0000 7f29da7a8640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:08:43.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:08:44.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:08:44.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:08:44.574 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:08:44.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:08:44.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:08:44.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:08:44.847 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/563274921,v1:127.0.0.1:6803/563274921] [v2:127.0.0.1:6804/563274921,v1:127.0.0.1:6805/563274921] exists,up c759759c-608d-48cc-b81b-07672690e9f7 2024-03-09T13:08:44.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:08:44.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:08:44.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:08:44.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-09T13:08:44.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T13:08:44.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:08:44.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:08:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:08:44.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:08:44.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T13:08:44.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:08:44.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:08:44.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ee104434-bea9-4a4d-b040-7458255d19ea 2024-03-09T13:08:44.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ee104434-bea9-4a4d-b040-7458255d19ea' 2024-03-09T13:08:44.854 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 ee104434-bea9-4a4d-b040-7458255d19ea 2024-03-09T13:08:44.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:08:44.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBcX+xlO675MxAAlk2C5kfQDlJXfTZLO2cdog== 2024-03-09T13:08:44.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBcX+xlO675MxAAlk2C5kfQDlJXfTZLO2cdog=="}' 2024-03-09T13:08:44.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ee104434-bea9-4a4d-b040-7458255d19ea -i td/osd-scrub-repair/1/new.json 2024-03-09T13:08:45.177 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:08:45.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:08:45.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBcX+xlO675MxAAlk2C5kfQDlJXfTZLO2cdog== --osd-uuid ee104434-bea9-4a4d-b040-7458255d19ea 2024-03-09T13:08:45.220 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:45.223+0000 7fe1b1c01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:45.245 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:45.248+0000 7fe1b1c01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:45.246 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:45.249+0000 7fe1b1c01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:45.246 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:45.249+0000 7fe1b1c01640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:08:45.247 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:45.249+0000 7fe1b1c01640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:08:45.247 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:45.249+0000 7fe1b1c01640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:08:45.337 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:45.340+0000 7fe1b1c01640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:08:51.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:08:51.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:08:51.128 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:08:51.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:08:51.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:08:51.486 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:08:51.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:08:51.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T13:08:51.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:08:51.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:08:51.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:08:51.515 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:51.518+0000 7fb29c00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:51.517 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:51.520+0000 7fb29c00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:51.517 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:51.521+0000 7fb29c00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:51.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:08:51.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:08:51.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:08:51.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:08:51.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:08:51.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:08:51.804 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:08:51.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:08:51.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:08:51.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:08:52.040 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:52.044+0000 7fb29c00d640 -1 Falling back to public interface 2024-03-09T13:08:52.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:08:53.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:08:53.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:08:53.076 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:08:53.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:08:53.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:08:53.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:08:53.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:08:54.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:08:54.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:08:54.365 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:08:54.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:08:54.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:08:54.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:08:54.509 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:54.513+0000 7fb29c00d640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:08:54.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:08:55.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:08:55.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:08:55.746 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:08:55.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:08:55.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:08:55.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:08:56.020 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1047511401,v1:127.0.0.1:6811/1047511401] [v2:127.0.0.1:6812/1047511401,v1:127.0.0.1:6813/1047511401] exists,up ee104434-bea9-4a4d-b040-7458255d19ea 2024-03-09T13:08:56.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:08:56.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:08:56.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:08:56.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2024-03-09T13:08:56.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:397: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T13:08:56.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:08:56.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:08:56.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:08:56.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:08:56.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:08:56.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:08:56.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T13:08:56.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:08:56.027 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:08:56.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=efae1f58-81b9-4a8a-9fe9-c02346247edc 2024-03-09T13:08:56.027 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 efae1f58-81b9-4a8a-9fe9-c02346247edc 2024-03-09T13:08:56.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 efae1f58-81b9-4a8a-9fe9-c02346247edc' 2024-03-09T13:08:56.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:08:56.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBoX+xlmUytAhAANRlgxo+1fciPqsJ/ZHZ4aw== 2024-03-09T13:08:56.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBoX+xlmUytAhAANRlgxo+1fciPqsJ/ZHZ4aw=="}' 2024-03-09T13:08:56.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new efae1f58-81b9-4a8a-9fe9-c02346247edc -i td/osd-scrub-repair/2/new.json 2024-03-09T13:08:56.463 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:08:56.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T13:08:56.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQBoX+xlmUytAhAANRlgxo+1fciPqsJ/ZHZ4aw== --osd-uuid efae1f58-81b9-4a8a-9fe9-c02346247edc 2024-03-09T13:08:56.506 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:56.510+0000 7f5a7c147640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:56.548 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:56.552+0000 7f5a7c147640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:56.549 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:56.553+0000 7f5a7c147640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:08:56.549 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:56.553+0000 7f5a7c147640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:08:56.549 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:56.553+0000 7f5a7c147640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:08:56.549 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:56.553+0000 7f5a7c147640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:08:56.615 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:08:56.618+0000 7f5a7c147640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T13:09:02.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T13:09:02.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:09:02.032 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T13:09:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T13:09:02.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:09:02.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T13:09:02.425 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T13:09:02.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T13:09:02.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:09:02.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:09:02.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:09:02.453 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:02.457+0000 7fdd10c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:09:02.455 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:02.459+0000 7fdd10c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:09:02.456 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:02.460+0000 7fdd10c0e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:09:02.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T13:09:02.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:09:02.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:09:02.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:09:02.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:09:02.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:02.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:09:02.705 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:09:02.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:02.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:09:02.972 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:02.976+0000 7fdd10c0e640 -1 Falling back to public interface 2024-03-09T13:09:02.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:09:03.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:09:03.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:03.981 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:09:03.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:09:03.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:03.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:09:04.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:09:05.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:09:05.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:05.262 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:09:05.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:09:05.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:05.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:09:05.281 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:05.284+0000 7fdd10c0e640 -1 osd.2 0 log_to_monitors true 2024-03-09T13:09:05.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:09:06.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:09:06.590 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:09:06.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:06.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:09:06.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:06.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:09:06.870 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/591599715,v1:127.0.0.1:6819/591599715] [v2:127.0.0.1:6820/591599715,v1:127.0.0.1:6821/591599715] exists,up efae1f58-81b9-4a8a-9fe9-c02346247edc 2024-03-09T13:09:06.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:09:06.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:09:06.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:09:06.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:400: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2024-03-09T13:09:06.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-09T13:09:07.233 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'testpool' created 2024-03-09T13:09:07.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:09:08.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2024-03-09T13:09:08.682 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 1 size to 2 2024-03-09T13:09:08.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:402: TEST_auto_repair_bluestore_tag: wait_for_clean 2024-03-09T13:09:08.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:09:08.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:09:08.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:09:08.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:09:08.694 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:09:08.694 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:09:08.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:09:08.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:09:08.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:09:08.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:09:08.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:09:08.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:09:08.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:09:08.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:09:08.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:09:09.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:09:09.054 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:09:09.054 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:09:09.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:09:09.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:09.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:09:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836486 2024-03-09T13:09:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836486 2024-03-09T13:09:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486' 2024-03-09T13:09:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:09.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:09:09.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672964 2024-03-09T13:09:09.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672964 2024-03-09T13:09:09.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486 1-42949672964' 2024-03-09T13:09:09.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:09.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:09:09.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509442 2024-03-09T13:09:09.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509442 2024-03-09T13:09:09.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836486 1-42949672964 2-64424509442' 2024-03-09T13:09:09.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:09.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836486 2024-03-09T13:09:09.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:09.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:09:09.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836486 2024-03-09T13:09:09.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:09.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836486 2024-03-09T13:09:09.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-03-09T13:09:09.528 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836486 2024-03-09T13:09:09.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:09:09.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836485 -lt 21474836486 2024-03-09T13:09:09.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:09:10.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:09:10.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:09:11.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836487 -lt 21474836486 2024-03-09T13:09:11.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:11.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672964 2024-03-09T13:09:11.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:11.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:09:11.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672964 2024-03-09T13:09:11.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:11.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672964 2024-03-09T13:09:11.088 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672964 2024-03-09T13:09:11.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-03-09T13:09:11.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:09:11.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672964 -lt 42949672964 2024-03-09T13:09:11.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:11.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509442 2024-03-09T13:09:11.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:11.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:09:11.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509442 2024-03-09T13:09:11.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:11.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509442 2024-03-09T13:09:11.364 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509442 2024-03-09T13:09:11.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2024-03-09T13:09:11.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:09:11.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509442 -lt 64424509442 2024-03-09T13:09:11.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:09:11.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:09:11.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:09:11.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:09:11.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:09:11.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:09:11.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:09:11.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:09:11.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:09:11.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:09:11.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:09:12.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:09:12.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:09:12.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:09:12.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:09:12.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:09:12.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:09:12.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:09:12.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:405: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2024-03-09T13:09:12.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: TEST_auto_repair_bluestore_tag: echo ABCDEF 2024-03-09T13:09:12.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:407: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T13:09:12.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-09T13:09:12.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T13:09:12.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:09:12.723 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T13:09:12.723 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:09:12.723 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:09:12.723 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:09:12.723 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:09:12.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T13:09:12.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:09:12.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T13:09:13.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:411: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:09:13.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:09:13.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:09:13.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:09:13.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T13:09:13.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:09:13.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:09:13.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:09:13.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:09:13.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:09:13.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-09T13:09:15.064 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-09T13:09:15.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T13:09:15.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:09:15.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:09:15.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:09:15.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:09:15.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:09:15.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:09:15.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:09:15.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:09:15.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:09:15.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:09:15.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:09:15.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:09:15.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:09:15.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:09:15.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:09:15.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:09:15.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:09:15.583 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:09:15.583 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:09:15.584 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:09:15.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:09:15.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:09:15.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:09:15.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:09:15.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:09:15.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:09:15.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:09:15.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:09:15.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:09:15.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:09:15.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T13:09:15.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:09:15.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:09:15.585 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:09:15.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T13:09:15.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:09:15.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:09:15.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:09:15.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:09:15.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:09:15.615 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:15.617+0000 7f04c9ec2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:09:15.616 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:15.619+0000 7f04c9ec2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:09:15.618 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:15.620+0000 7f04c9ec2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:09:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:09:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:09:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:09:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:09:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:09:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:09:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:09:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:15.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:09:16.132 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:16.135+0000 7f04c9ec2640 -1 Falling back to public interface 2024-03-09T13:09:16.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:09:17.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:09:17.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:17.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:09:17.141 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:09:17.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:17.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:09:17.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:09:18.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:09:18.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:18.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:09:18.417 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:09:18.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:18.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:09:18.578 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:18.581+0000 7f04c9ec2640 -1 osd.0 20 log_to_monitors true 2024-03-09T13:09:18.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:09:19.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:09:19.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:19.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:09:19.757 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:09:19.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:19.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:09:20.040 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/4175803246,v1:127.0.0.1:6803/4175803246] [v2:127.0.0.1:6804/4175803246,v1:127.0.0.1:6805/4175803246] exists,up c759759c-608d-48cc-b81b-07672690e9f7 2024-03-09T13:09:20.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:09:20.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:09:20.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:09:20.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:09:20.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:09:20.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:09:20.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:09:20.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:09:20.042 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:09:20.042 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:09:20.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:09:20.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:09:20.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:09:20.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:09:20.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:09:20.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:09:20.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:09:20.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:09:20.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:09:20.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:09:20.423 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:09:20.423 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:09:20.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:09:20.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:20.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:09:20.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215106 2024-03-09T13:09:20.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215106 2024-03-09T13:09:20.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106' 2024-03-09T13:09:20.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:20.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:09:20.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-03-09T13:09:20.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-03-09T13:09:20.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672967' 2024-03-09T13:09:20.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:20.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:09:20.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-03-09T13:09:20.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-03-09T13:09:20.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215106 1-42949672967 2-64424509445' 2024-03-09T13:09:20.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:20.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215106 2024-03-09T13:09:20.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:20.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:09:20.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215106 2024-03-09T13:09:20.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:20.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215106 2024-03-09T13:09:20.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2024-03-09T13:09:20.738 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 103079215106 2024-03-09T13:09:20.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:09:21.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-03-09T13:09:21.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:09:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:09:22.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:09:22.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 103079215106 2024-03-09T13:09:22.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:09:23.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:09:23.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:09:23.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215106 -lt 103079215106 2024-03-09T13:09:23.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:23.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-03-09T13:09:23.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:23.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:09:23.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-03-09T13:09:23.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:23.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-03-09T13:09:23.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-03-09T13:09:23.592 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672967 2024-03-09T13:09:23.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:09:23.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672967 2024-03-09T13:09:23.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:23.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-03-09T13:09:23.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:23.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:09:23.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-03-09T13:09:23.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:23.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-03-09T13:09:23.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-03-09T13:09:23.912 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509445 2024-03-09T13:09:23.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:09:24.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509445 2024-03-09T13:09:24.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:09:24.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:09:24.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:09:24.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:09:24.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:09:24.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:09:24.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:09:24.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:09:24.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:09:24.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:09:24.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:09:24.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:09:24.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:09:24.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:09:24.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:09:25.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:09:25.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:09:25.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:09:25.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2024-03-09T13:09:25.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T13:09:25.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:09:25.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool SOMETHING 2024-03-09T13:09:25.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:09:25.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:413: TEST_auto_repair_bluestore_tag: local pgid=1.0 2024-03-09T13:09:25.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2024-03-09T13:09:25.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:09:25.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:09:25.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:09:25.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:09:25.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:414: TEST_auto_repair_bluestore_tag: local primary=1 2024-03-09T13:09:25.738 INFO:tasks.workunit.client.0.smithi162.stdout:Affected PG 1.0 w/ primary 1 2024-03-09T13:09:25.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:415: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2024-03-09T13:09:25.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2024-03-09T13:09:25.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:09:25.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:09:25.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:09:25.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:09:26.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:416: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2024-03-09T13:09:07.188744+0000 2024-03-09T13:09:26.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:417: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2024-03-09T13:09:26.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: initiate_and_fetch_state: local the_osd=osd.1 2024-03-09T13:09:26.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: initiate_and_fetch_state: local pgid=1.0 2024-03-09T13:09:26.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2024-03-09T13:09:26.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T13:09:26.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:09:26.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:09:26.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T13:09:26.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:324: initiate_and_fetch_state: local last_scrub=2024-03-09T13:09:07.188744+0000 2024-03-09T13:09:26.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:326: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2024-03-09T13:09:26.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T13:09:26.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-09T13:09:26.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-03-09T13:09:26.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=3.0 2024-03-09T13:09:26.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T13:09:26.283 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-09T13:09:26.283 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T13:09:26.283 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T13:09:26.283 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:09:26.283 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:09:26.283 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:09:26.284 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.1.asok 2024-03-09T13:09:26.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70042/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2024-03-09T13:09:26.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T13:09:26.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:327: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2024-03-09T13:09:26.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T13:09:26.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-09T13:09:26.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_auto_repair 2024-03-09T13:09:26.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T13:09:26.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T13:09:26.372 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-09T13:09:26.372 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T13:09:26.373 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T13:09:26.373 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:09:26.373 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:09:26.373 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:09:26.373 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.1.asok 2024-03-09T13:09:26.373 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70042/ceph-osd.1.asok config set osd_scrub_auto_repair true 2024-03-09T13:09:26.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T13:09:26.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: initiate_and_fetch_state: flush_pg_stats 2024-03-09T13:09:26.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:09:26.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:09:26.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:09:26.732 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:09:26.732 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:09:26.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:09:26.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:26.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:09:26.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215108 2024-03-09T13:09:26.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215108 2024-03-09T13:09:26.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108' 2024-03-09T13:09:26.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:26.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:09:26.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672970 2024-03-09T13:09:26.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672970 2024-03-09T13:09:26.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108 1-42949672970' 2024-03-09T13:09:26.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:26.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:09:27.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-03-09T13:09:27.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-03-09T13:09:27.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215108 1-42949672970 2-64424509448' 2024-03-09T13:09:27.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:27.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215108 2024-03-09T13:09:27.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:27.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:09:27.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215108 2024-03-09T13:09:27.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:27.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215108 2024-03-09T13:09:27.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2024-03-09T13:09:27.046 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 103079215108 2024-03-09T13:09:27.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:09:27.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215107 -lt 103079215108 2024-03-09T13:09:27.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:09:28.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:09:28.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:09:28.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215108 -lt 103079215108 2024-03-09T13:09:28.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:28.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672970 2024-03-09T13:09:28.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:28.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:09:28.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672970 2024-03-09T13:09:28.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:28.607 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672970 2024-03-09T13:09:28.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672970 2024-03-09T13:09:28.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-03-09T13:09:28.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:09:28.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672970 2024-03-09T13:09:28.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:28.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-03-09T13:09:28.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:28.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:09:28.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-03-09T13:09:28.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:28.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-03-09T13:09:28.884 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509448 2024-03-09T13:09:28.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-03-09T13:09:28.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:09:29.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-03-09T13:09:29.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: initiate_and_fetch_state: date --rfc-3339=ns 2024-03-09T13:09:29.161 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09 13:09:29.165010075+00:00 2024-03-09T13:09:29.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: get_asok_path osd.1 2024-03-09T13:09:29.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T13:09:29.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T13:09:29.164 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:09:29.164 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:09:29.164 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:09:29.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.1.asok 2024-03-09T13:09:29.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70042/ceph-osd.1.asok schedule-deep-scrub 1.0 2024-03-09T13:09:29.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i=0 )) 2024-03-09T13:09:29.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-09T13:09:29.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-09T13:09:29.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-09T13:09:29.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-09T13:09:29.359 INFO:tasks.workunit.client.0.smithi162.stdout:{"deep":true,"must":false,"stamp":"2024-02-24T13:07:49.248643+0000"}0 ) state now: "active+clean" 2024-03-09T13:09:29.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2024-03-09T13:09:29.360 INFO:tasks.workunit.client.0.smithi162.stdout:"active+clean" 2024-03-09T13:09:29.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-09T13:09:29.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-09T13:09:29.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 0 == 4 ']' 2024-03-09T13:09:29.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-09T13:09:29.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-09T13:09:29.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-09T13:09:29.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-09T13:09:29.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-09T13:09:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-09T13:09:29.769 INFO:tasks.workunit.client.0.smithi162.stdout:1 ) state now: "active+clean" 2024-03-09T13:09:29.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2024-03-09T13:09:29.770 INFO:tasks.workunit.client.0.smithi162.stdout:"active+clean" 2024-03-09T13:09:29.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-09T13:09:29.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-09T13:09:29.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 1 == 4 ']' 2024-03-09T13:09:29.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-09T13:09:30.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-09T13:09:30.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-09T13:09:30.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-09T13:09:30.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-09T13:09:30.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean"' 2024-03-09T13:09:30.176 INFO:tasks.workunit.client.0.smithi162.stdout:2 ) state now: "active+clean" 2024-03-09T13:09:30.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2024-03-09T13:09:30.177 INFO:tasks.workunit.client.0.smithi162.stdout:"active+clean" 2024-03-09T13:09:30.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-09T13:09:30.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:346: initiate_and_fetch_state: echo '"active+clean"' 2024-03-09T13:09:30.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:349: initiate_and_fetch_state: '[' 2 == 4 ']' 2024-03-09T13:09:30.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: initiate_and_fetch_state: sleep 0.3 2024-03-09T13:09:30.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i++ )) 2024-03-09T13:09:30.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:336: initiate_and_fetch_state: (( i < 80 )) 2024-03-09T13:09:30.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: ceph pg 1.0 query --format json 2024-03-09T13:09:30.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: jq .state 2024-03-09T13:09:30.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:338: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2024-03-09T13:09:30.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean+scrubbing+deep"' 2024-03-09T13:09:30.584 INFO:tasks.workunit.client.0.smithi162.stdout:3 ) state now: "active+clean+scrubbing+deep" 2024-03-09T13:09:30.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: initiate_and_fetch_state: case "$st" in 2024-03-09T13:09:30.584 INFO:tasks.workunit.client.0.smithi162.stdout:found scrub 2024-03-09T13:09:30.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: echo 'found scrub' 2024-03-09T13:09:30.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: initiate_and_fetch_state: return 0 2024-03-09T13:09:30.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:418: TEST_auto_repair_bluestore_tag: r=0 2024-03-09T13:09:30.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:419: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2024-03-09T13:09:30.585 INFO:tasks.workunit.client.0.smithi162.stdout:initiate_and_fetch_state ret: 0 2024-03-09T13:09:30.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2024-03-09T13:09:30.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T13:09:30.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=td/osd-scrub-repair 2024-03-09T13:09:30.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_scrub_sleep 2024-03-09T13:09:30.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=0 2024-03-09T13:09:30.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T13:09:30.587 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.td/osd-scrub-repair 2024-03-09T13:09:30.587 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.td/osd-scrub-repair 2024-03-09T13:09:30.588 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2024-03-09T13:09:30.588 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:09:30.588 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:09:30.588 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:09:30.588 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.td/osd-scrub-repair.asok 2024-03-09T13:09:30.588 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70042/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2024-03-09T13:09:30.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test == true 2024-03-09T13:09:30.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1168: test: ==: unary operator expected 2024-03-09T13:09:30.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2024-03-09T13:09:30.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2024-03-09T13:09:30.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: wait_end_of_scrub: local the_osd=osd.1 2024-03-09T13:09:30.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:361: wait_end_of_scrub: local pgid=1.0 2024-03-09T13:09:30.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i=0 )) 2024-03-09T13:09:30.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-09T13:09:30.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-09T13:09:30.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-09T13:09:30.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-09T13:09:30.762 INFO:tasks.workunit.client.0.smithi162.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-09T13:09:30.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-09T13:09:30.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-09T13:09:30.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 0 == 4 ']' 2024-03-09T13:09:30.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-09T13:09:31.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-09T13:09:31.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-09T13:09:31.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-09T13:09:31.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-09T13:09:31.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-09T13:09:31.169 INFO:tasks.workunit.client.0.smithi162.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-09T13:09:31.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-09T13:09:31.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-09T13:09:31.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 1 == 4 ']' 2024-03-09T13:09:31.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-09T13:09:31.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-09T13:09:31.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-09T13:09:31.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-09T13:09:31.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-09T13:09:31.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-09T13:09:31.578 INFO:tasks.workunit.client.0.smithi162.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-09T13:09:31.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-09T13:09:31.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-09T13:09:31.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 2 == 4 ']' 2024-03-09T13:09:31.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-09T13:09:31.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-09T13:09:31.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-09T13:09:31.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-09T13:09:31.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-09T13:09:31.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-09T13:09:31.985 INFO:tasks.workunit.client.0.smithi162.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-09T13:09:31.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-09T13:09:31.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-09T13:09:31.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 3 == 4 ']' 2024-03-09T13:09:31.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-09T13:09:32.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-09T13:09:32.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-09T13:09:32.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-09T13:09:32.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-09T13:09:32.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2024-03-09T13:09:32.396 INFO:tasks.workunit.client.0.smithi162.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2024-03-09T13:09:32.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2024-03-09T13:09:32.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2024-03-09T13:09:32.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:367: wait_end_of_scrub: '[' 4 == 4 ']' 2024-03-09T13:09:32.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:368: wait_end_of_scrub: flush_pg_stats 2024-03-09T13:09:32.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:09:32.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:09:32.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:09:32.670 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:09:32.671 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:09:32.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:09:32.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:32.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:09:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=103079215110 2024-03-09T13:09:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 103079215110 2024-03-09T13:09:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110' 2024-03-09T13:09:32.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:32.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:09:32.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-03-09T13:09:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-03-09T13:09:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110 1-42949672972' 2024-03-09T13:09:32.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:32.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:09:32.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509450 2024-03-09T13:09:32.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509450 2024-03-09T13:09:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-103079215110 1-42949672972 2-64424509450' 2024-03-09T13:09:32.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:32.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-103079215110 2024-03-09T13:09:32.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:09:32.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-103079215110 2024-03-09T13:09:32.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=103079215110 2024-03-09T13:09:32.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 103079215110' 2024-03-09T13:09:32.985 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 103079215110 2024-03-09T13:09:32.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:09:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215109 -lt 103079215110 2024-03-09T13:09:33.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:09:34.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:09:34.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:09:34.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215111 -lt 103079215110 2024-03-09T13:09:34.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:34.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-03-09T13:09:34.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:34.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:09:34.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-03-09T13:09:34.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:34.533 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672972 2024-03-09T13:09:34.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-03-09T13:09:34.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-03-09T13:09:34.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:09:34.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-03-09T13:09:34.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:34.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509450 2024-03-09T13:09:34.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:34.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:09:34.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509450 2024-03-09T13:09:34.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:34.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509450 2024-03-09T13:09:34.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-03-09T13:09:34.807 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509450 2024-03-09T13:09:34.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:09:35.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509450 2024-03-09T13:09:35.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:370: wait_end_of_scrub: sleep 0.3 2024-03-09T13:09:35.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i++ )) 2024-03-09T13:09:35.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: wait_end_of_scrub: (( i < 40 )) 2024-03-09T13:09:35.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: ceph pg 1.0 query --format json 2024-03-09T13:09:35.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: jq .state 2024-03-09T13:09:35.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: wait_end_of_scrub: st='"active+clean"' 2024-03-09T13:09:35.532 INFO:tasks.workunit.client.0.smithi162.stdout:wait-scrub-end state now: "active+clean" 2024-03-09T13:09:35.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2024-03-09T13:09:35.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-03-09T13:09:35.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:366: wait_end_of_scrub: break 2024-03-09T13:09:35.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2024-03-09T13:09:35.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:378: wait_end_of_scrub: return 0 2024-03-09T13:09:35.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2024-03-09T13:09:35.787 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T13:09:35.787 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+scrubbing+deep 2024-03-09T13:09:30.432820+0000 20'1 25:42 [1,0] 1 [1,0] 1 0'0 2024-02-24T13:07:49.248643+0000 0'0 2024-02-24T13:07:49.248643+0000 0 0 deep scrubbing for 0s 0 0 2024-03-09T13:09:35.787 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T13:09:35.787 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T13:09:35.787 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T13:09:35.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-09T13:09:35.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T13:09:35.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:09:35.799 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T13:09:35.799 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:09:35.799 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:09:35.799 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:09:35.799 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:09:36.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T13:09:36.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:09:36.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T13:09:36.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T13:09:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:09:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:09:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:09:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:09:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T13:09:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:09:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:09:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:09:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:09:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:09:36.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:09:36.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:09:36.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:09:36.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:09:36.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:09:36.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:09:36.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2024-03-09T13:09:36.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:09:36.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:09:36.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:09:36.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:09:36.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:09:36.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2024-03-09T13:09:37.643 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:09:37.644 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:09:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T13:09:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:09:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:09:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:09:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:09:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:09:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:09:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:09:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:09:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:09:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:09:37.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:09:37.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:09:37.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:09:37.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:09:37.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:09:37.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:09:37.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:09:37.925 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:09:37.925 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:09:37.925 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:09:37.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:09:37.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:09:37.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:09:37.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:09:37.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:09:37.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:09:37.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:09:37.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:09:37.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:09:37.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:09:37.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T13:09:37.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:09:37.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:09:37.926 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:09:37.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T13:09:37.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:09:37.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:09:37.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:09:37.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:09:37.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:09:37.956 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:37.960+0000 7fc7b0ea4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:09:37.958 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:37.962+0000 7fc7b0ea4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:09:37.959 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:37.963+0000 7fc7b0ea4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:09:38.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:09:38.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:09:38.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:09:38.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:09:38.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:09:38.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:38.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:09:38.208 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:09:38.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:38.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:09:38.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:38.479+0000 7fc7b0ea4640 -1 Falling back to public interface 2024-03-09T13:09:38.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:09:39.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:09:39.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:39.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:09:39.506 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:09:39.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:39.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:09:39.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:09:40.737 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:40.741+0000 7fc7b0ea4640 -1 osd.0 25 log_to_monitors true 2024-03-09T13:09:40.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:09:40.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:40.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:09:40.783 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:09:40.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:40.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:09:41.153 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/1332212605,v1:127.0.0.1:6803/1332212605] [v2:127.0.0.1:6804/1332212605,v1:127.0.0.1:6805/1332212605] exists,up c759759c-608d-48cc-b81b-07672690e9f7 2024-03-09T13:09:41.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:09:41.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:09:41.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:09:41.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:09:41.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:09:41.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:09:41.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:09:41.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:09:41.154 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:09:41.154 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:09:41.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:09:41.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:09:41.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:09:41.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:09:41.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:09:41.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:09:41.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:09:41.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:09:41.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:09:41.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:09:41.543 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:09:41.543 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:09:41.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:09:41.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:41.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:09:41.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-03-09T13:09:41.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-03-09T13:09:41.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-03-09T13:09:41.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:41.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:09:41.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-09T13:09:41.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-09T13:09:41.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672975' 2024-03-09T13:09:41.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:41.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:09:41.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-09T13:09:41.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-09T13:09:41.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672975 2-64424509453' 2024-03-09T13:09:41.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:41.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-03-09T13:09:41.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:41.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:09:41.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-03-09T13:09:41.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:41.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-03-09T13:09:41.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-03-09T13:09:41.906 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 124554051586 2024-03-09T13:09:41.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:09:42.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 103079215111 -lt 124554051586 2024-03-09T13:09:42.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:09:43.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:09:43.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:09:43.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-03-09T13:09:43.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:43.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-09T13:09:43.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:43.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:09:43.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-09T13:09:43.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:43.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-09T13:09:43.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-09T13:09:43.500 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672975 2024-03-09T13:09:43.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:09:43.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-09T13:09:43.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:43.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-09T13:09:43.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:43.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:09:43.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-09T13:09:43.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:43.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-09T13:09:43.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-09T13:09:43.775 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509453 2024-03-09T13:09:43.775 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:09:44.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509453 -lt 64424509453 2024-03-09T13:09:44.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:09:44.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:09:44.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:09:44.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:09:44.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:09:44.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:09:44.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:09:44.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:09:44.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:09:44.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:09:44.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:09:44.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T13:09:44.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:09:44.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:09:44.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:09:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T13:09:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-09T13:09:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:09:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-09T13:09:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:09:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:09:45.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:09:45.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:09:45.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:09:45.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:09:45.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:09:45.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:09:45.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:09:45.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:09:45.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T13:09:45.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:09:45.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:09:45.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:09:45.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T13:09:45.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-03-09T13:09:45.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:09:45.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:09:45.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:09:45.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:09:45.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:09:45.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:09:45.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:09:46.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:09:46.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:09:46.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:09:46.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:09:46.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:09:46.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:09:46.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:09:46.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:09:46.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:09:46.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:09:46.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:09:46.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:09:46.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:09:46.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:09:46.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:09:46.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:09:46.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:09:46.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:09:46.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:09:46.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:09:46.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2024-03-09T13:09:46.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T13:09:46.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:09:46.929 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool SOMETHING 2024-03-09T13:09:46.929 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T13:09:46.929 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:09:46.929 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:09:46.929 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:09:47.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T13:09:47.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool SOMETHING 2024-03-09T13:09:47.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T13:09:47.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:431: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T13:09:47.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:09:47.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:09:47.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:09:47.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:09:47.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T13:09:47.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:09:47.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:09:47.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:09:47.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:09:47.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:09:47.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:09:47.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:09:47.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:09:47.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:09:47.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:09:47.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:09:47.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T13:09:47.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:09:47.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:09:47.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:09:47.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:09:47.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:09:47.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2024-03-09T13:09:49.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T13:09:49.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:09:49.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:09:49.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:09:49.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:09:49.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2024-03-09T13:09:49.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:09:49.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:09:49.308 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:09:49.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2024-03-09T13:09:49.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:09:49.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:09:49.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:09:49.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:09:49.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:09:49.336 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:49.340+0000 7f4e0eeac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:09:49.338 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:49.342+0000 7f4e0eeac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:09:49.340 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:49.343+0000 7f4e0eeac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:09:49.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:09:49.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:09:49.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:09:49.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:09:49.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:09:49.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:49.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:09:49.588 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:09:49.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:49.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:09:49.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:09:49.868 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:49.872+0000 7f4e0eeac640 -1 Falling back to public interface 2024-03-09T13:09:50.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:09:50.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:50.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:09:50.861 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:09:50.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:50.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:09:51.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:09:52.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:09:52.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:52.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:09:52.142 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:09:52.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:52.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:09:52.149 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:09:52.153+0000 7f4e0eeac640 -1 osd.0 30 log_to_monitors true 2024-03-09T13:09:52.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:09:53.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:09:53.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:09:53.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:09:53.459 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:09:53.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:09:53.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:09:53.738 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [29,30) [v2:127.0.0.1:6802/3339625591,v1:127.0.0.1:6803/3339625591] [v2:127.0.0.1:6804/3339625591,v1:127.0.0.1:6805/3339625591] exists,up c759759c-608d-48cc-b81b-07672690e9f7 2024-03-09T13:09:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:09:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:09:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:09:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:09:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:09:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:09:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:09:53.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:09:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:09:53.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:09:53.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:09:53.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:09:53.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:09:53.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:09:53.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:09:53.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:09:53.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:09:53.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:09:53.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:09:54.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:09:54.092 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:09:54.092 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:09:54.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:09:54.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:54.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:09:54.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=146028888066 2024-03-09T13:09:54.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 146028888066 2024-03-09T13:09:54.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066' 2024-03-09T13:09:54.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:54.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:09:54.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-09T13:09:54.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-09T13:09:54.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672978' 2024-03-09T13:09:54.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:09:54.302 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:09:54.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-09T13:09:54.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-09T13:09:54.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-146028888066 1-42949672978 2-64424509456' 2024-03-09T13:09:54.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:54.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-146028888066 2024-03-09T13:09:54.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:54.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:09:54.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-146028888066 2024-03-09T13:09:54.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:54.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=146028888066 2024-03-09T13:09:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2024-03-09T13:09:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 146028888066 2024-03-09T13:09:54.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:09:54.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 146028888066 -lt 146028888066 2024-03-09T13:09:54.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:54.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-09T13:09:54.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:54.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:09:54.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-09T13:09:54.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:54.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-09T13:09:54.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-09T13:09:54.686 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672978 2024-03-09T13:09:54.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:09:54.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672978 2024-03-09T13:09:54.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:09:55.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:09:55.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:09:56.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672978 2024-03-09T13:09:56.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:09:57.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:09:57.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:09:57.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672978 2024-03-09T13:09:57.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:09:57.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-09T13:09:57.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:09:57.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:09:57.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-09T13:09:57.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:09:57.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-09T13:09:57.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-09T13:09:57.531 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509456 2024-03-09T13:09:57.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:09:58.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509457 -lt 64424509456 2024-03-09T13:09:58.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:09:58.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:09:58.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:09:58.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T13:09:58.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:09:58.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:09:58.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:09:58.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:09:58.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:09:58.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:09:58.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:09:59.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T13:09:59.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:09:59.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:09:59.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:09:59.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T13:09:59.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:09:59.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:09:59.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:432: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:09:59.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2024-03-09T13:09:59.435 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:09:33.443+0000 7fb2821ec640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] planned AUTO_REPAIR TIME_FOR_DEEP] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2024-03-09T13:09:59.435 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:09:33.443+0000 7fb2821ec640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep [ 1.0: AUTO_REPAIR ] ] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2024-03-09T13:09:59.435 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:09:33.443+0000 7fb2821ec640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2024-03-09T13:09:59.435 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:09:33.443+0000 7fb2821ec640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2024-03-09T13:09:59.435 INFO:tasks.workunit.client.0.smithi162.stdout:2024-03-09T13:09:33.443+0000 7fb2821ec640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=17/17 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair [ 1.0: AUTO_REPAIR ] mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2024-03-09T13:09:59.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:09:59.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:09:59.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:09:59.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:09:59.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:09:59.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:09:59.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:09:59.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:09:59.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:09:59.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:09:59.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:09:59.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:09:59.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:09:59.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:09:59.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:09:59.546 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:09:59.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:09:59.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:09:59.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:09:59.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:09:59.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:09:59.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:09:59.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:09:59.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:09:59.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:09:59.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:09:59.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:09:59.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:09:59.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:09:59.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:09:59.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:09:59.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:09:59.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:09:59.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:09:59.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:09:59.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:09:59.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:09:59.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:09:59.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:09:59.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:09:59.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:09:59.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:09:59.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:09:59.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:09:59.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:09:59.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:09:59.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:09:59.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:09:59.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:09:59.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:09:59.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:09:59.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:09:59.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:09:59.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:09:59.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:09:59.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:09:59.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:09:59.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:09:59.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:09:59.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:09:59.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:09:59.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:09:59.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:09:59.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:09:59.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:09:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:09:59.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:09:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:09:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:09:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:09:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2024-03-09T13:09:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2024-03-09T13:09:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T13:09:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=false 2024-03-09T13:09:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-03-09T13:09:59.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-09T13:09:59.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:09:59.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:09:59.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:09:59.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:09:59.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:09:59.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T13:10:00.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:10:00.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:10:00.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:10:00.029 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:10:00.030 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:10:00.030 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:10:00.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:10:00.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T13:10:00.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:10:00.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:10:00.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:10:00.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:10:00.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:10:00.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:10:00.198 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:10:00.198 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:10:00.198 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:10:00.199 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:10:00.199 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:10:00.199 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:10:00.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:10:00.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:10:00.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:10:00.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:10:00.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:10:00.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:10:00.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:10:00.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:10:00.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:10:00.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:10:00.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:10:00.283 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:10:00.283 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:10:00.283 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:10:00.283 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:10:00.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:10:00.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:10:00.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-09T13:10:00.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:10:00.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:10:00.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:10:00.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:10:00.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:10:00.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:10:01.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:10:01.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:10:01.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:10:01.174 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:10:01.174 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:10:01.174 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:10:01.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:10:01.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:10:01.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:10:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:10:01.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-03-09T13:10:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-09T13:10:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:10:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:10:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:10:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:10:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:10:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:10:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:10:01.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:10:01.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:10:01.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:10:01.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c85c912c-3514-4e34-83a8-374c94c692ee 2024-03-09T13:10:01.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c85c912c-3514-4e34-83a8-374c94c692ee' 2024-03-09T13:10:01.210 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 c85c912c-3514-4e34-83a8-374c94c692ee 2024-03-09T13:10:01.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:10:01.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCpX+xlx5ilDRAAnH8aK/CGIhryf0wNsncg2Q== 2024-03-09T13:10:01.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCpX+xlx5ilDRAAnH8aK/CGIhryf0wNsncg2Q=="}' 2024-03-09T13:10:01.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c85c912c-3514-4e34-83a8-374c94c692ee -i td/osd-scrub-repair/0/new.json 2024-03-09T13:10:01.428 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:10:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:10:01.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCpX+xlx5ilDRAAnH8aK/CGIhryf0wNsncg2Q== --osd-uuid c85c912c-3514-4e34-83a8-374c94c692ee 2024-03-09T13:10:01.471 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:01.474+0000 7fdef9013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:10:01.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:01.482+0000 7fdef9013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:10:01.481 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:01.484+0000 7fdef9013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:10:01.481 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:01.484+0000 7fdef9013640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:10:01.481 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:01.484+0000 7fdef9013640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:10:01.481 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:01.484+0000 7fdef9013640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:10:01.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:01.566+0000 7fdef9013640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:10:07.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:10:07.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:10:07.102 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:10:07.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:10:07.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:10:07.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:10:07.461 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:10:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:10:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:10:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:10:07.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:10:07.490 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:07.493+0000 7f57cbd9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:10:07.492 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:07.495+0000 7f57cbd9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:10:07.493 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:07.496+0000 7f57cbd9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:10:07.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:10:07.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:10:07.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:10:07.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:10:07.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:10:07.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:07.737 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:10:07.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:10:07.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:07.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:08.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:08.020 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:08.023+0000 7f57cbd9a640 -1 Falling back to public interface 2024-03-09T13:10:09.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:09.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:09.010 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:10:09.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:10:09.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:09.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:09.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:10.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:10.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:10.293 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:10:10.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:10:10.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:10.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:10.376 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:10.379+0000 7f57cbd9a640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:10:10.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:11.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:11.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:11.576 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:10:11.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:10:11.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:11.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:11.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:12.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:12.861 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:10:12.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:10:12.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:12.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:13.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:14.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:14.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:14.163 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:10:14.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:10:14.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:14.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:14.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:15.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:15.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:15.448 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:10:15.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:10:15.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:15.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:15.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:16.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:16.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:16.734 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:10:16.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:10:16.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:16.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:17.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:18.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:18.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:18.019 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:10:18.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:10:18.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:18.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:18.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:19.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:19.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:19.346 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:10:19.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:10:19.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:19.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:19.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:20.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:20.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:20.630 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:10:20.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:10:20.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:20.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:20.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:21.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:21.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:21.991 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:10:21.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:10:21.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:21.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:22.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:23.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:23.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:23.604 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:10:23.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:10:23.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:23.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:24.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:25.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:25.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:25.112 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:10:25.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:10:25.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:25.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:26.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:26.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:26.874 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:10:26.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:10:26.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:26.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:27.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:28.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:28.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:28.284 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:10:28.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:10:28.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:28.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:28.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:29.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:29.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:29.954 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:10:29.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:10:29.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:29.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:30.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:31.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:31.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:31.647 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:10:31.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:10:31.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:31.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:32.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:33.019 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:10:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:10:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:33.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:10:33.300 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/527982104,v1:127.0.0.1:6803/527982104] [v2:127.0.0.1:6804/527982104,v1:127.0.0.1:6805/527982104] exists,up c85c912c-3514-4e34-83a8-374c94c692ee 2024-03-09T13:10:33.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:10:33.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:10:33.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:10:33.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-09T13:10:33.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:10:33.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:10:33.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:10:33.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:10:33.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:10:33.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:10:33.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:10:33.306 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:10:33.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:10:33.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:10:33.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:10:33.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:10:33.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:10:33.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:10:33.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:10:33.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:10:33.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:10:33.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:10:33.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:10:33.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:10:33.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:10:33.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:10:33.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:10:33.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=59358f2b-280f-4956-ac11-07775f645b8a 2024-03-09T13:10:33.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 59358f2b-280f-4956-ac11-07775f645b8a' 2024-03-09T13:10:33.307 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 59358f2b-280f-4956-ac11-07775f645b8a 2024-03-09T13:10:33.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:10:33.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJX+xl1VRaExAAAQ8PorZkLshrr/FC1zqR0A== 2024-03-09T13:10:33.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJX+xl1VRaExAAAQ8PorZkLshrr/FC1zqR0A=="}' 2024-03-09T13:10:33.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 59358f2b-280f-4956-ac11-07775f645b8a -i td/osd-scrub-repair/1/new.json 2024-03-09T13:10:33.659 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:10:33.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:10:33.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDJX+xl1VRaExAAAQ8PorZkLshrr/FC1zqR0A== --osd-uuid 59358f2b-280f-4956-ac11-07775f645b8a 2024-03-09T13:10:33.702 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:33.706+0000 7f79dc76d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:10:33.735 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:33.739+0000 7f79dc76d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:10:33.736 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:33.740+0000 7f79dc76d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:10:33.736 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:33.740+0000 7f79dc76d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:10:33.736 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:33.740+0000 7f79dc76d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:10:33.736 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:33.740+0000 7f79dc76d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:10:33.801 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:33.805+0000 7f79dc76d640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:10:39.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:10:39.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:10:39.257 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:10:39.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:10:39.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:10:39.612 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:10:39.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:10:39.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:10:39.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:10:39.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:10:39.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:10:39.641 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:39.645+0000 7f30dd818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:10:39.643 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:39.647+0000 7f30dd818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:10:39.644 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:39.648+0000 7f30dd818640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:10:39.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:10:39.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:10:39.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:10:39.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:10:39.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:10:39.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:39.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:10:39.885 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:10:39.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:39.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:40.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:40.171 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:40.175+0000 7f30dd818640 -1 Falling back to public interface 2024-03-09T13:10:41.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:41.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:41.158 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:10:41.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:10:41.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:41.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:41.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:42.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:42.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:42.445 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:10:42.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:10:42.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:42.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:42.541 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:10:42.545+0000 7f30dd818640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:10:42.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:43.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:43.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:43.750 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:10:43.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:10:43.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:43.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:44.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:45.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:45.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:45.035 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:10:45.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:10:45.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:45.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:45.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:46.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:46.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:46.428 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:10:46.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:10:46.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:46.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:46.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:47.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:47.712 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:10:47.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:47.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:10:47.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:47.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:48.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:49.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:49.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:49.060 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:10:49.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:10:49.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:49.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:49.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:50.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:50.478 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:10:50.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:50.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:10:50.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:50.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:50.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:51.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:51.763 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:10:51.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:51.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:10:51.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:51.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:52.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:53.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:53.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:53.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:10:53.046 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:10:53.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:53.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:53.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:54.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:54.369 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:10:54.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:54.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:10:54.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:54.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:54.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:55.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:55.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:55.674 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:10:55.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:10:55.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:55.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:56.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:57.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:57.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:57.165 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:10:57.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:10:57.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:57.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:57.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:10:58.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:10:58.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:10:58.550 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:10:58.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:10:58.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:10:58.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:10:59.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:00.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:00.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:00.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:11:00.220 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:11:00.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:00.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:11:00.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:01.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:01.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:01.893 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:11:01.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:11:01.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:01.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:11:02.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:03.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:03.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:03.626 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:11:03.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:11:03.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:03.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:11:04.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:05.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:05.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:05.353 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:11:05.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:11:05.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:05.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:11:05.675 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1432162291,v1:127.0.0.1:6811/1432162291] [v2:127.0.0.1:6812/1432162291,v1:127.0.0.1:6813/1432162291] exists,up 59358f2b-280f-4956-ac11-07775f645b8a 2024-03-09T13:11:05.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:11:05.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:11:05.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:11:05.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-09T13:11:05.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:11:05.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:11:05.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:11:05.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T13:11:05.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:11:05.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:11:05.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:11:05.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:11:05.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:11:05.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:11:05.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:11:05.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:11:05.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:11:05.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:11:05.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:11:05.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:11:05.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:11:05.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:11:05.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:11:05.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:11:05.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:11:05.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:11:05.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:11:05.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62c0d365-c223-4234-ad56-cf7eb13951a2 2024-03-09T13:11:05.683 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 62c0d365-c223-4234-ad56-cf7eb13951a2 2024-03-09T13:11:05.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 62c0d365-c223-4234-ad56-cf7eb13951a2' 2024-03-09T13:11:05.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:11:05.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDpX+xlKNe9KRAAQcbHilPBYzch//Rh7UdMcQ== 2024-03-09T13:11:05.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDpX+xlKNe9KRAAQcbHilPBYzch//Rh7UdMcQ=="}' 2024-03-09T13:11:05.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62c0d365-c223-4234-ad56-cf7eb13951a2 -i td/osd-scrub-repair/2/new.json 2024-03-09T13:11:06.015 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:11:06.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T13:11:06.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDpX+xlKNe9KRAAQcbHilPBYzch//Rh7UdMcQ== --osd-uuid 62c0d365-c223-4234-ad56-cf7eb13951a2 2024-03-09T13:11:06.058 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:06.060+0000 7f657ab8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:11:06.099 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:06.102+0000 7f657ab8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:11:06.100 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:06.103+0000 7f657ab8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:11:06.100 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:06.103+0000 7f657ab8c640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:11:06.100 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:06.103+0000 7f657ab8c640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:11:06.100 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:06.103+0000 7f657ab8c640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:11:06.174 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:06.177+0000 7f657ab8c640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T13:11:12.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T13:11:12.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:11:12.285 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T13:11:12.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T13:11:12.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:11:12.638 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T13:11:12.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T13:11:12.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:11:12.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:11:12.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:11:12.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:11:12.669 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:12.673+0000 7fb758af9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:11:12.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:12.675+0000 7fb758af9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:11:12.672 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:12.676+0000 7fb758af9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:11:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T13:11:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:11:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:11:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:11:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:11:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:12.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:11:12.913 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:11:12.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:12.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:13.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:13.189 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:13.193+0000 7fb758af9640 -1 Falling back to public interface 2024-03-09T13:11:14.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:14.186 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:11:14.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:14.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:11:14.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:14.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:14.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:15.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:15.469 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:11:15.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:15.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:11:15.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:15.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:15.640 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:15.644+0000 7fb758af9640 -1 osd.2 0 log_to_monitors true 2024-03-09T13:11:15.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:16.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:16.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:16.800 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:11:16.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:11:16.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:16.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:17.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:18.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:18.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:18.145 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:11:18.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:11:18.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:18.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:18.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:19.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:19.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:19.471 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:11:19.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:11:19.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:19.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:19.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:20.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:20.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:20.817 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:11:20.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:11:20.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:20.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:21.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:22.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:22.381 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:11:22.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:22.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:11:22.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:22.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:22.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:23.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:23.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:23.669 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:11:23.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:11:23.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:23.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:23.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:24.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:24.986 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:11:24.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:24.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:11:24.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:24.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:25.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:26.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:26.272 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:11:26.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:26.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:11:26.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:26.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:26.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:27.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:27.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:27.663 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:11:27.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:11:27.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:27.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:27.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:28.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:28.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:28.948 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:11:28.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:11:28.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:28.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:29.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:30.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:30.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:30.803 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:11:30.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:11:30.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:30.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:31.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:32.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:32.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:32.424 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:11:32.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:11:32.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:32.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:34.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:34.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:34.200 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:11:34.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:11:34.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:34.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:34.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:35.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:35.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:35.790 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:11:35.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:11:35.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:35.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:36.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:37.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:37.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:37.349 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:11:37.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:11:37.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:37.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:37.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:38.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:38.916 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:11:38.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:38.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:11:38.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:38.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:39.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:40.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:40.262 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:11:40.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:40.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:11:40.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:40.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:11:40.537 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/304506083,v1:127.0.0.1:6819/304506083] [v2:127.0.0.1:6820/304506083,v1:127.0.0.1:6821/304506083] exists,up 62c0d365-c223-4234-ad56-cf7eb13951a2 2024-03-09T13:11:40.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:11:40.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:11:40.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:11:40.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-03-09T13:11:40.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T13:11:40.801 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T13:11:40.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T13:11:40.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T13:11:41.179 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T13:11:41.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:11:42.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T13:11:44.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-03-09T13:11:44.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:11:44.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:11:44.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:11:44.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:11:44.244 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:11:44.244 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:11:44.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:11:44.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:11:44.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:11:44.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:11:44.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:11:44.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:11:44.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:11:44.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:11:44.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:11:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:11:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:11:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:11:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:11:44.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:11:44.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:11:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-03-09T13:11:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-03-09T13:11:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-03-09T13:11:44.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:11:44.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:11:44.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-09T13:11:44.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-09T13:11:44.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-64424509459' 2024-03-09T13:11:44.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:11:44.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:11:45.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=133143986182 2024-03-09T13:11:45.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 133143986182 2024-03-09T13:11:45.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-64424509459 2-133143986182' 2024-03-09T13:11:45.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:11:45.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-03-09T13:11:45.026 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:11:45.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:11:45.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-03-09T13:11:45.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:11:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-03-09T13:11:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-03-09T13:11:45.029 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836497 2024-03-09T13:11:45.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:11:45.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-03-09T13:11:45.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:11:46.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:11:46.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:11:46.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-03-09T13:11:46.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:11:46.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509459 2024-03-09T13:11:46.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:11:46.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:11:46.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509459 2024-03-09T13:11:46.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:11:46.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-09T13:11:46.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509459' 2024-03-09T13:11:46.632 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 64424509459 2024-03-09T13:11:46.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:11:46.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509459 2024-03-09T13:11:46.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:11:46.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-133143986182 2024-03-09T13:11:46.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:11:46.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:11:46.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-133143986182 2024-03-09T13:11:46.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:11:46.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=133143986182 2024-03-09T13:11:46.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 133143986182' 2024-03-09T13:11:46.916 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 133143986182 2024-03-09T13:11:46.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:11:47.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 133143986183 -lt 133143986182 2024-03-09T13:11:47.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:11:47.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:11:47.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:11:47.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T13:11:47.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:11:47.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:11:47.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:11:47.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:11:47.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:11:47.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:11:47.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:11:47.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:11:47.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:11:47.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:11:47.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:11:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T13:11:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:11:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:11:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-03-09T13:11:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T13:11:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T13:11:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-09T13:11:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T13:11:48.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-09T13:11:48.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T13:11:48.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T13:11:49.190 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' created 2024-03-09T13:11:49.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:11:50.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-09T13:11:50.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T13:11:50.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:11:50.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:11:50.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:11:50.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:11:50.205 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:11:50.205 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:11:50.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:11:50.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:11:50.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:11:50.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:11:50.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:11:50.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:11:50.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:11:50.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:11:50.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:11:50.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:11:50.578 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:11:50.578 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:11:50.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:11:50.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:11:50.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:11:50.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-09T13:11:50.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-09T13:11:50.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-09T13:11:50.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:11:50.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:11:50.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-03-09T13:11:50.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-03-09T13:11:50.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-64424509462' 2024-03-09T13:11:50.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:11:50.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:11:50.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=133143986185 2024-03-09T13:11:50.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 133143986185 2024-03-09T13:11:50.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-64424509462 2-133143986185' 2024-03-09T13:11:50.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:11:50.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-09T13:11:50.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:11:50.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:11:50.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-09T13:11:50.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:11:50.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-09T13:11:50.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-09T13:11:50.957 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836499 2024-03-09T13:11:50.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:11:51.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836499 2024-03-09T13:11:51.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:11:52.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:11:52.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:11:52.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-03-09T13:11:52.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:11:52.515 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-64424509462 2024-03-09T13:11:52.515 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:11:52.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:11:52.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-64424509462 2024-03-09T13:11:52.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:11:52.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-03-09T13:11:52.519 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 64424509462 2024-03-09T13:11:52.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 64424509462' 2024-03-09T13:11:52.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:11:52.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-03-09T13:11:52.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:11:52.792 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-133143986185 2024-03-09T13:11:52.792 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:11:52.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:11:52.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-133143986185 2024-03-09T13:11:52.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:11:52.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=133143986185 2024-03-09T13:11:52.795 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 133143986185 2024-03-09T13:11:52.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 133143986185' 2024-03-09T13:11:52.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:11:53.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 133143986185 -lt 133143986185 2024-03-09T13:11:53.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:11:53.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:11:53.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:11:53.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:11:53.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:11:53.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:11:53.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:11:53.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:11:53.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:11:53.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:11:53.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:11:53.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:11:53.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:11:53.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:11:53.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:11:54.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:11:54.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:11:54.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:11:54.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T13:11:54.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-03-09T13:11:54.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-03-09T13:11:54.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T13:11:54.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-09T13:11:54.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-09T13:11:54.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:11:54.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-09T13:11:54.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T13:11:54.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:11:54.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:11:54.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:11:54.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-09T13:11:54.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:11:54.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-09T13:11:54.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:11:54.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:11:54.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:11:54.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:11:54.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:11:54.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:11:54.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:11:54.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:11:54.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:11:54.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:11:54.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:11:54.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:11:54.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:11:54.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:11:54.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:11:54.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:11:55.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:11:55.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:11:55.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:11:55.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:11:55.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:11:55.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:11:55.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:11:55.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T13:11:56.629 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-09T13:11:56.629 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:11:57.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:11:57.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:11:57.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:11:57.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:11:57.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:11:57.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:11:57.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:11:57.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:11:57.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:11:57.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:11:57.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:11:57.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:11:57.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:11:57.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:11:57.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:11:57.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:11:57.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:11:57.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:11:57.149 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:11:57.149 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:11:57.149 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:11:57.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:11:57.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:11:57.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:11:57.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:11:57.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:11:57.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:11:57.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:11:57.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:11:57.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:11:57.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:11:57.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:11:57.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:11:57.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:11:57.150 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:11:57.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:11:57.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:11:57.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:11:57.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:11:57.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:11:57.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:11:57.180 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:57.183+0000 7fa380ed9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:11:57.182 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:57.185+0000 7fa380ed9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:11:57.183 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:57.186+0000 7fa380ed9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:11:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:11:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:11:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:11:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:11:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:11:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:57.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:11:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:11:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:57.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:11:57.705 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:11:57.708+0000 7fa380ed9640 -1 Falling back to public interface 2024-03-09T13:11:57.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:11:58.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:11:58.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:11:58.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:11:58.730 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:11:58.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:11:58.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:11:59.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:00.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:00.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:00.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:12:00.019 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:12:00.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:00.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:12:00.094 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:00.097+0000 7fa380ed9640 -1 osd.1 52 log_to_monitors true 2024-03-09T13:12:00.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:01.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:01.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:01.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:12:01.308 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:12:01.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:01.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:12:01.800 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 53 last_clean_interval [15,52) [v2:127.0.0.1:6810/1068189619,v1:127.0.0.1:6811/1068189619] [v2:127.0.0.1:6812/1068189619,v1:127.0.0.1:6813/1068189619] exists,up 59358f2b-280f-4956-ac11-07775f645b8a 2024-03-09T13:12:01.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:12:01.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:12:01.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:12:01.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:12:01.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:12:01.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:12:01.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:12:01.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:12:01.800 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:12:01.801 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:12:01.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:12:01.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:12:01.801 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:12:01.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:12:01.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:12:01.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:12:01.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:12:01.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:12:01.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:12:02.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:12:02.227 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:12:02.227 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:12:02.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:12:02.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:12:02.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:12:02.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-03-09T13:12:02.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-03-09T13:12:02.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-03-09T13:12:02.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:12:02.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:12:02.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103170 2024-03-09T13:12:02.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103170 2024-03-09T13:12:02.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-249108103170' 2024-03-09T13:12:02.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:12:02.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:12:02.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=133143986188 2024-03-09T13:12:02.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 133143986188 2024-03-09T13:12:02.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-249108103170 2-133143986188' 2024-03-09T13:12:02.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:12:02.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-03-09T13:12:02.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:12:02.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:12:02.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-03-09T13:12:02.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:12:02.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-03-09T13:12:02.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-03-09T13:12:02.643 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836502 2024-03-09T13:12:02.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:12:02.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-03-09T13:12:02.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:12:03.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:12:03.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:12:04.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836502 2024-03-09T13:12:04.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:12:04.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103170 2024-03-09T13:12:04.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:12:04.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:12:04.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103170 2024-03-09T13:12:04.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:12:04.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103170 2024-03-09T13:12:04.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103170' 2024-03-09T13:12:04.262 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 249108103170 2024-03-09T13:12:04.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:12:04.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103170 -lt 249108103170 2024-03-09T13:12:04.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:12:04.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-133143986188 2024-03-09T13:12:04.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:12:04.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:12:04.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-133143986188 2024-03-09T13:12:04.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:12:04.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=133143986188 2024-03-09T13:12:04.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 133143986188' 2024-03-09T13:12:04.585 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 133143986188 2024-03-09T13:12:04.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:12:04.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 133143986188 -lt 133143986188 2024-03-09T13:12:04.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:12:04.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:12:04.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:12:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:12:05.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:12:05.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:12:05.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:12:05.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:12:05.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:12:05.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:12:05.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:12:05.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:12:05.530 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:12:05.530 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:12:05.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:12:05.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:12:05.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:12:05.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:12:05.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-03-09T13:12:05.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:12:05.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:12:05.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:12:05.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:12:06.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-03-09T13:12:06.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-03-09T13:12:06.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:12:06.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:12:06.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:12:06.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:12:06.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-03-09T13:12:01.826880+0000 2024-03-09T13:12:06.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:12:06.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:12:01.826880+0000 2024-03-09T13:12:06.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:12:06.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:12:06.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:12:06.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:12:06.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:12:06.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:12:06.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:12:06.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:12:06.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:12:01.826880+0000 '>' 2024-03-09T13:12:01.826880+0000 2024-03-09T13:12:06.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:12:07.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:12:07.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:12:07.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:12:07.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:12:07.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:12:07.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:12:07.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:12:07.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:12:01.826880+0000 '>' 2024-03-09T13:12:01.826880+0000 2024-03-09T13:12:07.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:12:08.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:12:08.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:12:09.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:12:09.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:12:09.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:12:09.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:12:09.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:12:09.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:12:01.826880+0000 '>' 2024-03-09T13:12:01.826880+0000 2024-03-09T13:12:09.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:12:10.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:12:10.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:12:10.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:12:10.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:12:10.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:12:10.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:12:10.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:12:10.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:12:01.826880+0000 '>' 2024-03-09T13:12:01.826880+0000 2024-03-09T13:12:10.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:12:11.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:12:11.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:12:11.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:12:11.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:12:11.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:12:11.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:12:11.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:12:11.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:12:09.651872+0000 '>' 2024-03-09T13:12:01.826880+0000 2024-03-09T13:12:11.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:12:11.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-03-09T13:12:11.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:12:11.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:12:11.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:12:11.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:12:11.856 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:12:11.856 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:12:11.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:12:11.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:12:11.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:12:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:12:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:12:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:12:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:12:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:12:11.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:12:12.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:12:12.229 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:12:12.229 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:12:12.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:12:12.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:12:12.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:12:12.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-03-09T13:12:12.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-03-09T13:12:12.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-03-09T13:12:12.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:12:12.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:12:12.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103173 2024-03-09T13:12:12.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103173 2024-03-09T13:12:12.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-249108103173' 2024-03-09T13:12:12.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:12:12.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:12:12.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=133143986191 2024-03-09T13:12:12.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 133143986191 2024-03-09T13:12:12.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-249108103173 2-133143986191' 2024-03-09T13:12:12.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:12:12.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-03-09T13:12:12.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:12:12.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:12:12.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-03-09T13:12:12.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:12:12.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-03-09T13:12:12.551 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836505 2024-03-09T13:12:12.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-03-09T13:12:12.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:12:12.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-03-09T13:12:12.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:12:13.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:12:13.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:12:14.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836505 2024-03-09T13:12:14.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:12:14.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103173 2024-03-09T13:12:14.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:12:14.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:12:14.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103173 2024-03-09T13:12:14.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:12:14.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103173 2024-03-09T13:12:14.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103173' 2024-03-09T13:12:14.145 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 249108103173 2024-03-09T13:12:14.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:12:14.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103173 -lt 249108103173 2024-03-09T13:12:14.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:12:14.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-133143986191 2024-03-09T13:12:14.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:12:14.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:12:14.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-133143986191 2024-03-09T13:12:14.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:12:14.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=133143986191 2024-03-09T13:12:14.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 133143986191' 2024-03-09T13:12:14.422 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 133143986191 2024-03-09T13:12:14.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:12:14.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 133143986191 -lt 133143986191 2024-03-09T13:12:14.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:12:14.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:12:14.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:12:15.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:12:15.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:12:15.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:12:15.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:12:15.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:12:15.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:12:15.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:12:15.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:12:15.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:12:15.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:12:15.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:12:15.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:12:15.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:12:15.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:12:15.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:12:15.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-09T13:12:15.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-09T13:12:15.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:12:15.730 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-09T13:12:15.730 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T13:12:15.731 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:12:15.731 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:12:15.731 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:12:16.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-09T13:12:16.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:12:16.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-09T13:12:16.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:12:16.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:12:16.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:12:16.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:12:16.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:12:16.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:12:16.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:12:16.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:12:16.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:12:16.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:12:16.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:12:16.285 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:12:16.285 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:12:16.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:12:16.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:12:16.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:12:16.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:12:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:12:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:12:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:12:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:12:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:12:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:12:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T13:12:18.332 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-09T13:12:18.332 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-09T13:12:18.332 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:12:18.332 INFO:tasks.workunit.client.0.smithi162.stdout:hinfo_key 2024-03-09T13:12:18.332 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:12:18.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:12:18.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:12:18.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:12:18.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:12:18.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:12:18.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:12:18.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:12:18.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:12:18.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:12:18.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:12:18.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:12:18.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:12:18.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:12:18.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:12:18.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:12:18.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:12:18.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:12:18.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:12:18.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:12:18.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:12:18.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:12:18.602 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:12:18.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:12:18.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:12:18.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:12:18.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:12:18.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:12:18.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:12:18.633 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:18.637+0000 7f323616d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:12:18.635 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:18.639+0000 7f323616d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:12:18.636 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:18.640+0000 7f323616d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:12:18.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:12:18.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:12:18.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:12:18.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:12:18.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:12:18.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:18.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:12:18.885 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:12:18.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:18.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:12:19.154 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:19.158+0000 7f323616d640 -1 Falling back to public interface 2024-03-09T13:12:19.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:20.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:20.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:20.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:12:20.162 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:12:20.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:20.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:12:20.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:21.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:21.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:21.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:12:21.444 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:12:21.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:21.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:12:21.573 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:21.577+0000 7f323616d640 -1 osd.1 68 log_to_monitors true 2024-03-09T13:12:21.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:22.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:22.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:22.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:12:22.769 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:12:22.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:22.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:12:23.100 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 75 up_thru 75 down_at 69 last_clean_interval [58,68) [v2:127.0.0.1:6810/2488068576,v1:127.0.0.1:6811/2488068576] [v2:127.0.0.1:6812/2488068576,v1:127.0.0.1:6813/2488068576] exists,up 59358f2b-280f-4956-ac11-07775f645b8a 2024-03-09T13:12:23.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:12:23.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:12:23.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:12:23.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:12:23.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:12:23.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:12:23.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:12:23.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:12:23.101 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:12:23.101 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:12:23.102 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:12:23.102 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:12:23.102 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:12:23.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:12:23.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:12:23.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:12:23.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:12:23.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:12:23.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:12:23.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:12:23.478 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:12:23.478 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:12:23.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:12:23.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:12:23.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:12:23.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-03-09T13:12:23.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-03-09T13:12:23.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-03-09T13:12:23.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:12:23.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:12:23.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-03-09T13:12:23.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-03-09T13:12:23.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-322122547202' 2024-03-09T13:12:23.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:12:23.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:12:23.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=133143986194 2024-03-09T13:12:23.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 133143986194 2024-03-09T13:12:23.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-322122547202 2-133143986194' 2024-03-09T13:12:23.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:12:23.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-03-09T13:12:23.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:12:23.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:12:23.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-03-09T13:12:23.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:12:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-03-09T13:12:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-03-09T13:12:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836509 2024-03-09T13:12:23.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:12:24.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-03-09T13:12:24.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:12:25.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:12:25.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:12:25.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-03-09T13:12:25.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:12:25.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-322122547202 2024-03-09T13:12:25.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:12:25.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:12:25.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-322122547202 2024-03-09T13:12:25.386 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:12:25.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-03-09T13:12:25.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 322122547202' 2024-03-09T13:12:25.387 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 322122547202 2024-03-09T13:12:25.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:12:25.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547202 -lt 322122547202 2024-03-09T13:12:25.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:12:25.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-133143986194 2024-03-09T13:12:25.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:12:25.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:12:25.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-133143986194 2024-03-09T13:12:25.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:12:25.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=133143986194 2024-03-09T13:12:25.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 133143986194' 2024-03-09T13:12:25.667 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 133143986194 2024-03-09T13:12:25.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:12:25.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 133143986195 -lt 133143986194 2024-03-09T13:12:25.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:12:25.942 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:12:25.942 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:12:26.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:12:26.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:12:26.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:12:26.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:12:26.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:12:26.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:12:26.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:12:26.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:12:26.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:12:26.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:12:26.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:12:26.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:12:26.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:12:26.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:12:26.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:12:26.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:12:26.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:12:26.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:12:26.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:12:26.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:12:26.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:12:26.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:12:26.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:12:26.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:12:26.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:12:26.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:12:27.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:12:27.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:12:27.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:12:27.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:12:27.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:12:27.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:12:27.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:12:27.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:12:27.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:12:27.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:12:27.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:12:27.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:12:27.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:12:27.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:12:27.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:12:27.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:12:27.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:12:27.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:12:27.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:12:27.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:12:27.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:12:27.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:12:27.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:12:27.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:12:27.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:12:27.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:12:27.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:12:27.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:12:27.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:12:27.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:12:27.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:12:27.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:12:27.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:12:27.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:12:27.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:12:27.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:12:27.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:12:27.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:12:27.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:12:27.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:12:27.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:12:27.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:12:27.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:12:27.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:12:27.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:12:27.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:12:27.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:12:27.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:12:27.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:12:27.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:12:27.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:12:27.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:12:27.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:12:27.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:12:27.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:12:27.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:12:27.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:12:27.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:12:27.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:12:27.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:12:27.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:12:27.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2024-03-09T13:12:27.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2024-03-09T13:12:27.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2024-03-09T13:12:27.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T13:12:27.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: auto_repair_erasure_coded: local allow_overwrites=true 2024-03-09T13:12:27.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: auto_repair_erasure_coded: local poolname=ecpool 2024-03-09T13:12:27.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-09T13:12:27.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:12:27.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:12:27.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:12:27.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:12:27.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:12:27.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T13:12:27.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:12:27.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:12:27.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:12:27.476 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:12:27.476 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:12:27.476 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:12:27.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:12:27.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T13:12:27.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:12:27.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:12:27.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:12:27.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:12:27.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:12:27.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:12:27.651 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:12:27.651 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:12:27.651 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:12:27.651 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:12:27.652 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:12:27.652 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:12:27.652 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:12:27.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:12:27.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:12:27.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:12:27.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:12:27.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:12:27.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:12:27.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:12:27.731 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:12:27.732 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:12:27.732 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:12:27.732 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:12:27.732 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:12:27.732 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:12:27.732 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:12:27.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:12:27.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:12:27.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:274: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-09T13:12:27.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:12:27.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:12:27.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:12:27.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:12:27.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:12:27.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:12:28.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:12:28.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:12:28.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:12:28.617 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:12:28.617 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:12:28.617 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:12:28.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:12:28.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:12:28.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:12:28.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:279: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:12:28.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: seq 0 2 2024-03-09T13:12:28.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-09T13:12:28.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:12:28.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:12:28.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:12:28.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:12:28.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:12:28.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:12:28.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:12:28.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:12:28.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:12:28.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:12:28.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:12:28.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:12:28.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:12:28.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:12:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:12:28.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:12:28.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:12:28.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:12:28.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:12:28.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:12:28.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:12:28.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:12:28.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:12:28.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8f7f88e9-d54e-4af8-8f6c-944a1b36afa4 2024-03-09T13:12:28.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8f7f88e9-d54e-4af8-8f6c-944a1b36afa4' 2024-03-09T13:12:28.653 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 8f7f88e9-d54e-4af8-8f6c-944a1b36afa4 2024-03-09T13:12:28.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:12:28.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8YOxlJW4HKBAAYJtLf3e2qk+bjPWmk6r4xA== 2024-03-09T13:12:28.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8YOxlJW4HKBAAYJtLf3e2qk+bjPWmk6r4xA=="}' 2024-03-09T13:12:28.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8f7f88e9-d54e-4af8-8f6c-944a1b36afa4 -i td/osd-scrub-repair/0/new.json 2024-03-09T13:12:28.873 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:12:28.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:12:28.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQA8YOxlJW4HKBAAYJtLf3e2qk+bjPWmk6r4xA== --osd-uuid 8f7f88e9-d54e-4af8-8f6c-944a1b36afa4 2024-03-09T13:12:28.916 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:28.919+0000 7fe27e0a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:12:28.924 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:28.927+0000 7fe27e0a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:12:28.925 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:28.928+0000 7fe27e0a4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:12:28.925 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:28.928+0000 7fe27e0a4640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:12:28.926 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:28.929+0000 7fe27e0a4640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:12:28.926 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:28.929+0000 7fe27e0a4640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:12:29.016 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:29.019+0000 7fe27e0a4640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:12:34.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:12:34.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:12:34.499 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:12:34.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:12:34.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:12:34.857 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:12:34.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:12:34.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:12:34.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:12:34.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:12:34.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:12:34.887 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:34.890+0000 7fb635a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:12:34.889 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:34.892+0000 7fb635a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:12:34.890 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:34.893+0000 7fb635a14640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:12:35.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:12:35.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:12:35.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:12:35.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:12:35.176 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:12:35.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:12:35.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:35.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:12:35.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:35.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:35.409 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:35.412+0000 7fb635a14640 -1 Falling back to public interface 2024-03-09T13:12:35.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:36.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:36.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:36.452 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:12:36.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:12:36.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:36.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:36.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:37.704 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:12:37.707+0000 7fb635a14640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:12:37.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:37.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:37.728 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:12:37.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:12:37.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:37.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:38.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:39.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:39.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:39.139 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:12:39.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:12:39.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:39.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:39.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:40.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:40.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:40.495 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:12:40.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:12:40.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:40.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:40.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:41.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:41.777 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:12:41.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:41.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:12:41.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:41.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:42.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:43.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:43.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:43.060 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:12:43.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:12:43.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:43.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:43.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:44.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:44.475 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:12:44.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:44.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:12:44.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:44.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:44.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:45.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:45.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:45.760 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:12:45.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:12:45.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:45.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:46.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:47.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:47.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:47.046 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:12:47.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:12:47.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:47.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:47.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:48.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:48.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:48.423 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:12:48.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:12:48.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:48.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:48.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:49.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:49.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:49.708 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:12:49.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:12:49.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:49.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:50.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:51.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:51.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:51.256 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:12:51.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:12:51.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:51.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:51.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:52.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:52.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:52.917 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:12:52.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:12:52.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:52.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:53.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:54.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:54.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:54.770 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:12:54.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:12:54.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:54.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:55.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:56.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:56.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:56.236 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:12:56.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:12:56.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:56.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:56.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:57.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:57.814 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:12:57.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:57.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:12:57.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:57.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:58.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:12:59.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:12:59.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:12:59.417 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:12:59.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:12:59.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:12:59.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:12:59.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:00.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:00.713 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:13:00.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:00.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:13:00.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:00.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:13:00.998 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3646048142,v1:127.0.0.1:6803/3646048142] [v2:127.0.0.1:6804/3646048142,v1:127.0.0.1:6805/3646048142] exists,up 8f7f88e9-d54e-4af8-8f6c-944a1b36afa4 2024-03-09T13:13:01.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:13:01.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:13:01.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:13:01.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-09T13:13:01.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:13:01.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:13:01.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:13:01.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:13:01.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:13:01.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:13:01.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:13:01.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:13:01.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:13:01.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:13:01.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c6c5a53-6ab8-448b-b3e4-89f1ea1eb2fc 2024-03-09T13:13:01.007 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 8c6c5a53-6ab8-448b-b3e4-89f1ea1eb2fc 2024-03-09T13:13:01.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8c6c5a53-6ab8-448b-b3e4-89f1ea1eb2fc' 2024-03-09T13:13:01.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:13:01.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBdYOxl7pVmARAA5nVBChEQjMpFZyCf4EQ7QQ== 2024-03-09T13:13:01.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBdYOxl7pVmARAA5nVBChEQjMpFZyCf4EQ7QQ=="}' 2024-03-09T13:13:01.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c6c5a53-6ab8-448b-b3e4-89f1ea1eb2fc -i td/osd-scrub-repair/1/new.json 2024-03-09T13:13:01.394 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:13:01.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:13:01.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQBdYOxl7pVmARAA5nVBChEQjMpFZyCf4EQ7QQ== --osd-uuid 8c6c5a53-6ab8-448b-b3e4-89f1ea1eb2fc 2024-03-09T13:13:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:01.442+0000 7fc4e4c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:13:01.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:01.483+0000 7fc4e4c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:13:01.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:01.484+0000 7fc4e4c22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:13:01.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:01.484+0000 7fc4e4c22640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:13:01.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:01.484+0000 7fc4e4c22640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:13:01.480 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:01.484+0000 7fc4e4c22640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:13:01.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:01.566+0000 7fc4e4c22640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:13:07.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:13:07.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:13:07.186 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:13:07.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:13:07.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:13:07.545 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:13:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:13:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:13:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:13:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:13:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:13:07.576 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:07.580+0000 7f1f40cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:13:07.578 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:07.582+0000 7f1f40cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:13:07.579 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:07.583+0000 7f1f40cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:13:07.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:13:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:13:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:13:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:13:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:13:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:07.820 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:13:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:13:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:08.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:08.117 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:08.121+0000 7f1f40cb5640 -1 Falling back to public interface 2024-03-09T13:13:09.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:09.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:09.092 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:13:09.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:13:09.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:09.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:09.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:10.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:10.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:10.372 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:13:10.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:13:10.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:10.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:10.519 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:10.522+0000 7f1f40cb5640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:13:10.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:11.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:11.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:11.712 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:13:11.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:13:11.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:11.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:11.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:12.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:12.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:12.996 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:13:12.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:13:12.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:12.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:13.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:14.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:14.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:14.280 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:13:14.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:13:14.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:14.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:14.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:15.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:15.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:15.564 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:13:15.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:13:15.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:15.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:15.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:16.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:16.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:16.848 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:13:16.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:13:16.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:16.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:17.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:18.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:18.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:18.259 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:13:18.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:13:18.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:18.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:18.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:19.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:19.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:19.544 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:13:19.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:13:19.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:19.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:19.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:20.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:20.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:20.823 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:13:20.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:13:20.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:20.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:21.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:22.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:22.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:22.109 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:13:22.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:13:22.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:22.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:22.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:23.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:23.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:23.392 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:13:23.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:13:23.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:23.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:24.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:25.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:25.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:25.239 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:13:25.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:13:25.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:25.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:25.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:26.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:26.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:26.804 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:13:26.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:13:26.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:26.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:27.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:28.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:28.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:28.828 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:13:28.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:13:28.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:28.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:29.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:30.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:30.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:30.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:13:30.589 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:13:30.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:30.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:31.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:32.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:32.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:32.067 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:13:32.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:13:32.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:32.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:32.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:33.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:33.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:33.961 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:13:33.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:13:33.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:33.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:13:34.241 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 16 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2885693598,v1:127.0.0.1:6811/2885693598] [v2:127.0.0.1:6812/2885693598,v1:127.0.0.1:6813/2885693598] exists,up 8c6c5a53-6ab8-448b-b3e4-89f1ea1eb2fc 2024-03-09T13:13:34.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:13:34.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:13:34.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:13:34.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:280: auto_repair_erasure_coded: for id in $(seq 0 2) 2024-03-09T13:13:34.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:281: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:13:34.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:13:34.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:13:34.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T13:13:34.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:13:34.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:13:34.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:13:34.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:13:34.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:13:34.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:13:34.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:13:34.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:13:34.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:13:34.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:13:34.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:13:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:13:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:13:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:13:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:13:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:13:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:13:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:13:34.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:13:34.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=560ca052-b002-4275-981d-4b9e7fb40788 2024-03-09T13:13:34.248 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 560ca052-b002-4275-981d-4b9e7fb40788 2024-03-09T13:13:34.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 560ca052-b002-4275-981d-4b9e7fb40788' 2024-03-09T13:13:34.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:13:34.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+YOxlP6neDxAAusgNJsar+55FNAExawF1oQ== 2024-03-09T13:13:34.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+YOxlP6neDxAAusgNJsar+55FNAExawF1oQ=="}' 2024-03-09T13:13:34.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 560ca052-b002-4275-981d-4b9e7fb40788 -i td/osd-scrub-repair/2/new.json 2024-03-09T13:13:34.611 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:13:34.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T13:13:34.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQB+YOxlP6neDxAAusgNJsar+55FNAExawF1oQ== --osd-uuid 560ca052-b002-4275-981d-4b9e7fb40788 2024-03-09T13:13:34.655 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:34.659+0000 7ff88be11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:13:34.721 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:34.724+0000 7ff88be11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:13:34.722 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:34.726+0000 7ff88be11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:13:34.723 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:34.726+0000 7ff88be11640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:13:34.723 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:34.726+0000 7ff88be11640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:13:34.723 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:34.726+0000 7ff88be11640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:13:34.788 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:34.791+0000 7ff88be11640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T13:13:40.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T13:13:40.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:13:40.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T13:13:40.933 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T13:13:40.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:13:41.291 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T13:13:41.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T13:13:41.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:13:41.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:13:41.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:13:41.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:13:41.320 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:41.324+0000 7faf9a224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:13:41.322 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:41.326+0000 7faf9a224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:13:41.323 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:41.327+0000 7faf9a224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:13:41.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T13:13:41.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:13:41.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:13:41.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:13:41.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:13:41.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:41.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:13:41.566 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:13:41.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:41.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:41.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:41.843 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:41.848+0000 7faf9a224640 -1 Falling back to public interface 2024-03-09T13:13:42.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:42.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:42.839 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:13:42.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:13:42.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:42.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:43.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:44.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:44.121 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:13:44.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:44.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:13:44.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:44.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:44.154 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:13:44.158+0000 7faf9a224640 -1 osd.2 0 log_to_monitors true 2024-03-09T13:13:44.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:45.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:45.505 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:13:45.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:45.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:13:45.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:45.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:45.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:46.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:46.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:46.872 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:13:46.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:13:46.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:46.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:47.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:48.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:48.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:48.299 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:13:48.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:13:48.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:48.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:48.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:49.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:49.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:49.584 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:13:49.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:13:49.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:49.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:49.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:50.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:50.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:50.871 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:13:50.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:13:50.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:50.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:52.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:52.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:52.160 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:13:52.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:13:52.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:52.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:52.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:53.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:53.442 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:13:53.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:53.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:13:53.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:53.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:53.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:54.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:54.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:54.787 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:13:54.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:13:54.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:54.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:55.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:56.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:56.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:56.114 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:13:56.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:13:56.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:56.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:56.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:57.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:57.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:57.509 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:13:57.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:13:57.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:57.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:57.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:13:58.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:13:58.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:13:58.870 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:13:58.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:13:58.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:13:58.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:13:59.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:14:00.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:14:00.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:14:00.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:14:00.823 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:14:00.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:14:00.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:14:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:14:02.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:14:02.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:14:02.205 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:14:02.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:14:02.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:14:02.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:14:03.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:14:04.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:14:04.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:14:04.347 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:14:04.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:14:04.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:14:04.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:14:04.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:14:05.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:14:05.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:14:05.799 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:14:05.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:14:05.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:14:05.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:14:06.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:14:07.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:14:07.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:14:07.264 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:14:07.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:14:07.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:14:07.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:14:07.546 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3245504862,v1:127.0.0.1:6819/3245504862] [v2:127.0.0.1:6820/3245504862,v1:127.0.0.1:6821/3245504862] exists,up 560ca052-b002-4275-981d-4b9e7fb40788 2024-03-09T13:14:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:14:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:14:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:14:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:283: auto_repair_erasure_coded: create_rbd_pool 2024-03-09T13:14:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T13:14:07.812 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T13:14:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T13:14:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T13:14:08.323 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T13:14:08.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:14:09.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T13:14:11.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:284: auto_repair_erasure_coded: wait_for_clean 2024-03-09T13:14:11.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:14:11.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:14:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:14:11.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:14:11.213 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:14:11.214 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:14:11.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:14:11.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:14:11.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:14:11.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:14:11.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:14:11.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:14:11.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:14:11.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:14:11.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:14:11.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:14:11.581 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:14:11.582 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:14:11.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:14:11.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:11.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:14:11.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-03-09T13:14:11.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-03-09T13:14:11.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-03-09T13:14:11.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:11.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:14:11.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476753 2024-03-09T13:14:11.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476753 2024-03-09T13:14:11.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-68719476753' 2024-03-09T13:14:11.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:11.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:14:11.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920777 2024-03-09T13:14:11.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920777 2024-03-09T13:14:11.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-68719476753 2-141733920777' 2024-03-09T13:14:11.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:11.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-03-09T13:14:11.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:11.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:14:11.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-03-09T13:14:11.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:11.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-03-09T13:14:11.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-03-09T13:14:11.895 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836497 2024-03-09T13:14:11.895 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:14:12.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-03-09T13:14:12.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:14:13.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:14:13.165 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:14:13.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-03-09T13:14:13.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:13.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476753 2024-03-09T13:14:13.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:13.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:14:13.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476753 2024-03-09T13:14:13.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:13.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476753 2024-03-09T13:14:13.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476753' 2024-03-09T13:14:13.458 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 68719476753 2024-03-09T13:14:13.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:14:13.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476753 -lt 68719476753 2024-03-09T13:14:13.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:13.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920777 2024-03-09T13:14:13.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:13.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:14:13.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920777 2024-03-09T13:14:13.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:13.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920777 2024-03-09T13:14:13.764 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 141733920777 2024-03-09T13:14:13.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920777' 2024-03-09T13:14:13.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:14:14.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920777 -lt 141733920777 2024-03-09T13:14:14.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:14:14.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:14:14.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:14:14.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T13:14:14.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:14:14.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:14:14.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:14:14.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:14:14.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:14:14.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:14:14.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:14:14.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:14:14.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:14:14.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:14:14.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:14:15.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T13:14:15.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:14:15.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:14:15.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-03-09T13:14:15.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T13:14:15.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T13:14:15.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-09T13:14:15.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T13:14:15.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-09T13:14:15.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T13:14:15.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T13:14:16.297 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' created 2024-03-09T13:14:16.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:14:17.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-09T13:14:17.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-09T13:14:17.686 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 2 allow_ec_overwrites to true 2024-03-09T13:14:17.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T13:14:17.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:14:17.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:14:17.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:14:17.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:14:17.698 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:14:17.698 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:14:17.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:14:17.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:14:17.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:14:17.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:14:17.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:14:17.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:14:17.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:14:17.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:14:17.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:14:18.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:14:18.268 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:14:18.268 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:14:18.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:14:18.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:18.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:14:18.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836499 2024-03-09T13:14:18.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836499 2024-03-09T13:14:18.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499' 2024-03-09T13:14:18.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:18.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:14:18.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=68719476755 2024-03-09T13:14:18.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 68719476755 2024-03-09T13:14:18.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-68719476755' 2024-03-09T13:14:18.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:18.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:14:18.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920780 2024-03-09T13:14:18.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920780 2024-03-09T13:14:18.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836499 1-68719476755 2-141733920780' 2024-03-09T13:14:18.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:18.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836499 2024-03-09T13:14:18.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:18.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:14:18.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836499 2024-03-09T13:14:18.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:18.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836499 2024-03-09T13:14:18.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-03-09T13:14:18.698 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836499 2024-03-09T13:14:18.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:14:19.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836499 2024-03-09T13:14:19.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:19.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-68719476755 2024-03-09T13:14:19.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:19.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:14:19.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-68719476755 2024-03-09T13:14:19.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:19.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=68719476755 2024-03-09T13:14:19.060 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 68719476755 2024-03-09T13:14:19.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 68719476755' 2024-03-09T13:14:19.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:14:19.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 68719476755 -lt 68719476755 2024-03-09T13:14:19.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:19.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920780 2024-03-09T13:14:19.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:19.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:14:19.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920780 2024-03-09T13:14:19.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:19.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920780 2024-03-09T13:14:19.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920780' 2024-03-09T13:14:19.379 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 141733920780 2024-03-09T13:14:19.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:14:19.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920779 -lt 141733920780 2024-03-09T13:14:19.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:14:20.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:14:20.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:14:20.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920780 -lt 141733920780 2024-03-09T13:14:20.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:14:20.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:14:20.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:14:21.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:14:21.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:14:21.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:14:21.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:14:21.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:14:21.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:14:21.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:14:21.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:14:21.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:14:21.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:14:21.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:14:21.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:14:21.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:14:21.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:14:21.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:14:21.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T13:14:21.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:290: auto_repair_erasure_coded: local payload=ABCDEF 2024-03-09T13:14:21.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:291: auto_repair_erasure_coded: echo ABCDEF 2024-03-09T13:14:21.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:292: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T13:14:22.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-09T13:14:22.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-09T13:14:22.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:14:22.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-09T13:14:22.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T13:14:22.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:14:22.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:14:22.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:14:22.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-09T13:14:22.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:14:22.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-09T13:14:22.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:296: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:14:22.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:14:22.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:14:22.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:14:22.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:14:22.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:14:22.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:14:22.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:14:22.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:14:22.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:14:22.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:14:22.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:14:22.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:14:22.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:14:22.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:14:22.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:14:24.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:14:24.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:14:24.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:14:24.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:14:24.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:14:24.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:14:24.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:14:24.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T13:14:25.552 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-09T13:14:25.552 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:14:26.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:14:26.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:14:26.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:14:26.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:14:26.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:14:26.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:14:26.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:14:26.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:14:26.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:14:26.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:14:26.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:14:26.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:14:26.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:14:26.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:14:26.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:14:26.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:14:26.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:14:26.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:14:26.070 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:14:26.070 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:14:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:14:26.072 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:14:26.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:14:26.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:14:26.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:14:26.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:14:26.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:14:26.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:14:26.102 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:26.106+0000 7ff1bc1aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:14:26.104 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:26.109+0000 7ff1bc1aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:14:26.106 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:26.110+0000 7ff1bc1aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:14:26.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:14:26.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:14:26.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:14:26.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:14:26.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:14:26.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:14:26.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:14:26.353 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:14:26.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:14:26.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:14:26.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:14:26.625 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:26.630+0000 7ff1bc1aa640 -1 Falling back to public interface 2024-03-09T13:14:27.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:14:27.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:14:27.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:14:27.626 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:14:27.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:14:27.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:14:27.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:14:28.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:14:28.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:14:28.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:14:28.910 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:14:28.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:14:28.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:14:29.106 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:29.110+0000 7ff1bc1aa640 -1 osd.1 54 log_to_monitors true 2024-03-09T13:14:29.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:14:30.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:14:30.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:14:30.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:14:30.212 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:14:30.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:14:30.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:14:30.539 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 61 up_thru 61 down_at 55 last_clean_interval [16,54) [v2:127.0.0.1:6810/1573419456,v1:127.0.0.1:6811/1573419456] [v2:127.0.0.1:6812/1573419456,v1:127.0.0.1:6813/1573419456] exists,up 8c6c5a53-6ab8-448b-b3e4-89f1ea1eb2fc 2024-03-09T13:14:30.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:14:30.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:14:30.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:14:30.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:14:30.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:14:30.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:14:30.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:14:30.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:14:30.540 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:14:30.540 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:14:30.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:14:30.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:14:30.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:14:30.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:14:30.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:14:30.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:14:30.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:14:30.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:14:30.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:14:31.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:14:31.006 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:14:31.006 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:14:31.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:14:31.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:31.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:14:31.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-09T13:14:31.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-09T13:14:31.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-09T13:14:31.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:31.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:14:31.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005058 2024-03-09T13:14:31.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005058 2024-03-09T13:14:31.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-261993005058' 2024-03-09T13:14:31.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:31.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:14:31.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920783 2024-03-09T13:14:31.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920783 2024-03-09T13:14:31.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-261993005058 2-141733920783' 2024-03-09T13:14:31.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:31.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-09T13:14:31.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:31.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:14:31.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-09T13:14:31.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:31.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-09T13:14:31.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-09T13:14:31.405 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836503 2024-03-09T13:14:31.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:14:31.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-09T13:14:31.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:14:32.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:14:32.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:14:32.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-09T13:14:32.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:32.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-261993005058 2024-03-09T13:14:32.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:32.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:14:32.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-261993005058 2024-03-09T13:14:32.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005058 2024-03-09T13:14:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 261993005058' 2024-03-09T13:14:32.975 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 261993005058 2024-03-09T13:14:32.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:14:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005058 -lt 261993005058 2024-03-09T13:14:33.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:33.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920783 2024-03-09T13:14:33.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:33.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:14:33.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920783 2024-03-09T13:14:33.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920783 2024-03-09T13:14:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920783' 2024-03-09T13:14:33.263 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 141733920783 2024-03-09T13:14:33.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:14:33.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920783 -lt 141733920783 2024-03-09T13:14:33.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:14:33.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:14:33.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:14:33.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:14:33.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:14:33.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:14:33.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:14:33.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:14:33.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:14:33.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:14:33.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:14:34.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:14:34.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:14:34.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:14:34.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:14:34.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:14:34.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:14:34.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:14:34.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2024-03-09T13:14:34.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:14:34.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:14:34.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:14:34.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:14:34.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: auto_repair_erasure_coded: local pgid=2.0 2024-03-09T13:14:34.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2024-03-09T13:14:34.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:14:34.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:14:34.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:14:34.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:14:35.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:299: auto_repair_erasure_coded: wait_for_scrub 2.0 2024-03-09T13:14:21.920548+0000 2024-03-09T13:14:35.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:14:35.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:14:21.920548+0000 2024-03-09T13:14:35.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:14:35.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:14:35.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:14:35.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:14:35.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:14:35.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:14:35.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:14:35.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:14:35.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:14:21.920548+0000 '>' 2024-03-09T13:14:21.920548+0000 2024-03-09T13:14:35.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:14:36.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:14:36.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:14:36.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:14:36.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:14:36.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:14:36.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:14:36.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:14:36.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:14:21.920548+0000 '>' 2024-03-09T13:14:21.920548+0000 2024-03-09T13:14:36.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:14:37.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:14:37.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:14:37.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:14:37.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:14:37.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:14:37.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:14:37.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:14:38.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:14:21.920548+0000 '>' 2024-03-09T13:14:21.920548+0000 2024-03-09T13:14:38.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:14:39.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:14:39.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:14:39.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:14:39.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:14:39.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:14:39.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:14:39.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:14:39.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:14:36.334431+0000 '>' 2024-03-09T13:14:21.920548+0000 2024-03-09T13:14:39.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:14:39.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:300: auto_repair_erasure_coded: wait_for_clean 2024-03-09T13:14:39.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:14:39.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:14:39.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:14:39.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:14:39.379 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:14:39.379 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:14:39.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:14:39.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:14:39.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:14:39.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:14:39.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:14:39.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:14:39.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:14:39.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:14:39.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:14:39.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:14:39.758 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:14:39.758 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:14:39.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:14:39.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:39.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:14:39.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-03-09T13:14:39.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-03-09T13:14:39.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-03-09T13:14:39.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:39.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:14:39.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005061 2024-03-09T13:14:39.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005061 2024-03-09T13:14:39.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-261993005061' 2024-03-09T13:14:39.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:39.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:14:40.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920786 2024-03-09T13:14:40.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920786 2024-03-09T13:14:40.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-261993005061 2-141733920786' 2024-03-09T13:14:40.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:40.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-03-09T13:14:40.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:40.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:14:40.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-03-09T13:14:40.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:40.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-03-09T13:14:40.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-03-09T13:14:40.083 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836505 2024-03-09T13:14:40.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:14:40.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-03-09T13:14:40.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:14:41.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:14:41.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:14:41.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836505 2024-03-09T13:14:41.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:41.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-261993005061 2024-03-09T13:14:41.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:41.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:14:41.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-261993005061 2024-03-09T13:14:41.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:41.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005061 2024-03-09T13:14:41.650 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 261993005061 2024-03-09T13:14:41.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 261993005061' 2024-03-09T13:14:41.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:14:41.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005061 -lt 261993005061 2024-03-09T13:14:41.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:41.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920786 2024-03-09T13:14:41.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:41.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:14:41.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920786 2024-03-09T13:14:41.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:41.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920786 2024-03-09T13:14:41.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920786' 2024-03-09T13:14:41.928 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 141733920786 2024-03-09T13:14:41.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:14:42.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920786 -lt 141733920786 2024-03-09T13:14:42.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:14:42.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:14:42.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:14:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:14:42.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:14:42.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:14:42.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:14:42.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:14:42.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:14:42.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:14:42.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:14:42.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:14:42.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:14:42.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:14:42.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:14:43.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:14:43.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:14:43.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:14:43.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2024-03-09T13:14:43.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=ecpool 2024-03-09T13:14:43.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:14:43.158 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary ecpool SOMETHING 2024-03-09T13:14:43.159 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T13:14:43.159 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:14:43.159 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:14:43.159 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:14:43.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-03-09T13:14:43.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:14:43.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-03-09T13:14:43.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:14:43.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:14:43.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:14:43.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:14:43.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:14:43.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:14:43.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:14:43.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:14:43.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:14:43.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:14:43.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:14:43.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:14:43.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:14:43.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:14:43.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:14:43.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:14:44.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:14:44.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:14:44.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:14:44.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:14:44.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:14:44.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:14:44.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:14:44.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T13:14:45.684 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-09T13:14:45.684 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2024-03-09T13:14:45.684 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:14:45.684 INFO:tasks.workunit.client.0.smithi162.stdout:hinfo_key 2024-03-09T13:14:45.684 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:14:45.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:14:45.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:14:45.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:14:45.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:14:45.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:14:45.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:14:45.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:14:45.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:14:45.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:14:45.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:14:45.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:14:45.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:14:45.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:14:45.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:14:45.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:14:45.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:14:45.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:14:45.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:14:45.956 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:14:45.957 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:14:45.957 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:14:45.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:14:45.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:14:45.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:14:45.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:14:45.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:14:45.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:14:45.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:14:45.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:14:45.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:14:45.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:14:45.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T13:14:45.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:14:45.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:14:45.958 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:14:45.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2024-03-09T13:14:45.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:14:45.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:14:45.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:14:45.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:14:45.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:14:45.988 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:45.991+0000 7f5799a12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:14:46.001 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:46.004+0000 7f5799a12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:14:46.002 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:46.005+0000 7f5799a12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:14:46.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:14:46.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:14:46.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:14:46.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:14:46.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:14:46.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:14:46.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:14:46.267 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:14:46.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:14:46.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:14:46.519 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:46.522+0000 7f5799a12640 -1 Falling back to public interface 2024-03-09T13:14:46.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:14:47.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:14:47.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:14:47.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:14:47.541 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:14:47.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:14:47.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:14:47.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:14:48.836 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:48.839+0000 7f5799a12640 -1 osd.1 70 log_to_monitors true 2024-03-09T13:14:48.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:14:48.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:14:48.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:14:48.838 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:14:48.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:14:48.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:14:49.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:14:50.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:14:50.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:14:50.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:14:50.250 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:14:50.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:14:50.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:14:50.888 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 71 last_clean_interval [61,70) [v2:127.0.0.1:6810/2384229074,v1:127.0.0.1:6811/2384229074] [v2:127.0.0.1:6812/2384229074,v1:127.0.0.1:6813/2384229074] exists,up 8c6c5a53-6ab8-448b-b3e4-89f1ea1eb2fc 2024-03-09T13:14:50.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:14:50.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:14:50.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:14:50.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:14:50.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:14:50.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:14:50.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:14:50.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:14:50.889 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:14:50.889 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:14:50.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:14:50.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:14:50.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:14:50.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:14:50.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:14:50.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:14:50.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:14:50.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:14:50.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:14:51.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:14:51.271 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:14:51.271 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:14:51.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:14:51.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:51.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:14:51.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-03-09T13:14:51.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-03-09T13:14:51.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-03-09T13:14:51.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:51.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:14:51.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-09T13:14:51.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-09T13:14:51.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-330712481794' 2024-03-09T13:14:51.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:14:51.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:14:51.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=141733920789 2024-03-09T13:14:51.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 141733920789 2024-03-09T13:14:51.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-330712481794 2-141733920789' 2024-03-09T13:14:51.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:51.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-03-09T13:14:51.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:51.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:14:51.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-03-09T13:14:51.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:51.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-03-09T13:14:51.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-03-09T13:14:51.588 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836509 2024-03-09T13:14:51.588 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:14:51.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-03-09T13:14:51.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:14:52.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:14:52.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:14:53.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-03-09T13:14:53.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:53.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-03-09T13:14:53.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:53.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:14:53.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-03-09T13:14:53.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:53.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-09T13:14:53.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-03-09T13:14:53.147 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 330712481794 2024-03-09T13:14:53.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:14:53.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481794 -lt 330712481794 2024-03-09T13:14:53.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:14:53.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-141733920789 2024-03-09T13:14:53.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:14:53.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:14:53.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-141733920789 2024-03-09T13:14:53.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:14:53.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=141733920789 2024-03-09T13:14:53.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 141733920789' 2024-03-09T13:14:53.423 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 141733920789 2024-03-09T13:14:53.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:14:53.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 141733920789 -lt 141733920789 2024-03-09T13:14:53.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:14:53.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:14:53.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:14:54.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:14:54.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:14:54.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:14:54.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:14:54.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:14:54.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:14:54.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:14:54.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:14:54.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:14:54.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:14:54.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:14:54.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:14:54.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:14:54.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:14:54.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:14:54.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:14:54.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:305: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:14:54.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:14:54.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:14:54.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:14:54.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:14:54.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:14:54.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:14:54.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:14:54.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:14:54.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:14:54.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:14:54.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:14:54.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:14:54.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:14:54.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:14:54.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:14:54.811 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:14:54.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:14:54.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:14:54.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:14:54.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:14:54.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:14:54.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:14:54.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:14:54.831 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:14:54.831 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:14:54.831 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:14:54.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:14:54.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:14:54.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:14:54.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:14:54.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:14:54.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:14:54.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:14:54.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:14:54.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:14:54.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:14:54.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:14:54.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:14:54.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:14:54.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:14:54.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:14:54.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:14:54.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:14:54.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:14:54.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:14:54.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:14:54.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:14:54.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:14:54.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:14:54.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:14:54.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:14:54.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:14:54.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:14:54.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:14:54.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:14:54.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:14:54.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:14:54.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:14:54.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:14:54.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:14:54.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:14:54.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:14:54.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:14:54.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:14:54.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:14:54.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:14:54.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:14:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:14:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:14:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:14:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2024-03-09T13:14:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2024-03-09T13:14:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-03-09T13:14:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=false 2024-03-09T13:14:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-03-09T13:14:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-03-09T13:14:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:14:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:14:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:14:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:14:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:14:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T13:14:55.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:14:55.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:14:55.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:14:55.303 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:14:55.303 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:14:55.303 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:14:55.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:14:55.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T13:14:55.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:14:55.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:14:55.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:14:55.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:14:55.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:14:55.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:14:55.480 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:14:55.480 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:14:55.480 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:14:55.480 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:14:55.480 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:14:55.480 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:14:55.480 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:14:55.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:14:55.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:14:55.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:14:55.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:14:55.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:14:55.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:14:55.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:14:55.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:14:55.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:14:55.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:14:55.564 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:14:55.564 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:14:55.564 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:14:55.564 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:14:55.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:14:55.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:14:55.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-03-09T13:14:55.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:14:55.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:14:55.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:14:55.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:14:55.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:14:55.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:14:56.447 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:14:56.447 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:14:56.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:14:56.448 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:14:56.448 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:14:56.448 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:14:56.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:14:56.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:14:56.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:14:56.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-03-09T13:14:56.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T13:14:56.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-03-09T13:14:56.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:14:56.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:14:56.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:14:56.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:14:56.481 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:14:56.481 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:14:56.481 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:14:56.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:14:56.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:14:56.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:14:56.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:14:56.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:14:56.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:14:56.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:14:56.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:14:56.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:14:56.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:14:56.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:14:56.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:14:56.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:14:56.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:14:56.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:14:56.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50eeb74e-7873-40ad-a32c-09e6e8458ce1 2024-03-09T13:14:56.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 50eeb74e-7873-40ad-a32c-09e6e8458ce1' 2024-03-09T13:14:56.482 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 50eeb74e-7873-40ad-a32c-09e6e8458ce1 2024-03-09T13:14:56.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:14:56.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQYOxlQtPdHRAAlhawWcXGSp558qd2RkRxkg== 2024-03-09T13:14:56.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQYOxlQtPdHRAAlhawWcXGSp558qd2RkRxkg=="}' 2024-03-09T13:14:56.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50eeb74e-7873-40ad-a32c-09e6e8458ce1 -i td/osd-scrub-repair/0/new.json 2024-03-09T13:14:56.736 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:14:56.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:14:56.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQYOxlQtPdHRAAlhawWcXGSp558qd2RkRxkg== --osd-uuid 50eeb74e-7873-40ad-a32c-09e6e8458ce1 2024-03-09T13:14:56.775 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:56.778+0000 7f8f0ac17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:14:56.778 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:56.781+0000 7f8f0ac17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:14:56.779 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:56.782+0000 7f8f0ac17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:14:56.780 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:56.783+0000 7f8f0ac17640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:14:56.780 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:56.783+0000 7f8f0ac17640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:14:56.780 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:56.783+0000 7f8f0ac17640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:14:56.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:14:56.865+0000 7f8f0ac17640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:15:02.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:15:02.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:15:02.289 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:15:02.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:15:02.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:15:02.648 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:15:02.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:15:02.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:15:02.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:15:02.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:15:02.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:15:02.674 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:02.677+0000 7fe6334a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:15:02.702 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:02.704+0000 7fe6334a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:15:02.703 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:02.706+0000 7fe6334a8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:15:02.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:15:02.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:15:02.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:15:02.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:15:02.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:15:02.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:02.964 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:15:02.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:15:02.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:02.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:03.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:03.247 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:03.250+0000 7fe6334a8640 -1 Falling back to public interface 2024-03-09T13:15:04.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:04.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:04.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:15:04.237 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:15:04.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:04.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:04.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:05.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:05.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:05.519 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:15:05.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:15:05.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:05.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:05.535 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:05.538+0000 7fe6334a8640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:15:05.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:06.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:06.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:06.946 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:15:06.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:15:06.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:06.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:07.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:08.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:08.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:08.229 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:15:08.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:15:08.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:08.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:08.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:09.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:09.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:09.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:15:09.512 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:15:09.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:09.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:09.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:10.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:10.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:10.794 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:15:10.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:15:10.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:10.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:11.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:12.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:12.078 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:15:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:15:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:12.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:12.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:13.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:13.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:13.445 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:15:13.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:15:13.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:13.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:13.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:14.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:14.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:14.726 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:15:14.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:15:14.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:14.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:15.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:16.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:16.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:16.085 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:15:16.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:15:16.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:16.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:16.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:17.818 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:15:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:15:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:18.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:19.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:19.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:19.278 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:15:19.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:15:19.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:19.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:19.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:20.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:20.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:20.849 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:15:20.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:15:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:20.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:21.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:22.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:22.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:22.706 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:15:22.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:15:22.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:22.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:23.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:24.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:24.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:24.374 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:15:24.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:15:24.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:24.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:24.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:25.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:25.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:25.744 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:15:25.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:15:25.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:25.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:26.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:27.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:27.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:27.414 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:15:27.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:15:27.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:27.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:15:27.696 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3523272541,v1:127.0.0.1:6803/3523272541] [v2:127.0.0.1:6804/3523272541,v1:127.0.0.1:6805/3523272541] exists,up 50eeb74e-7873-40ad-a32c-09e6e8458ce1 2024-03-09T13:15:27.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:15:27.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:15:27.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:15:27.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T13:15:27.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-03-09T13:15:27.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:15:27.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:15:27.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:15:27.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:15:27.702 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:15:27.702 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:15:27.702 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:15:27.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:15:27.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:15:27.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:15:27.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:15:27.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:15:27.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:15:27.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:15:27.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:15:27.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:15:27.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:15:27.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:15:27.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:15:27.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:15:27.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:15:27.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:15:27.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=55ce8a92-5cd3-49b9-823d-afa9a56afea9 2024-03-09T13:15:27.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 55ce8a92-5cd3-49b9-823d-afa9a56afea9' 2024-03-09T13:15:27.703 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 55ce8a92-5cd3-49b9-823d-afa9a56afea9 2024-03-09T13:15:27.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:15:27.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvYOxlDz3yKhAArWPrpQnBZdTVoTztepIW0Q== 2024-03-09T13:15:27.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvYOxlDz3yKhAArWPrpQnBZdTVoTztepIW0Q=="}' 2024-03-09T13:15:27.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 55ce8a92-5cd3-49b9-823d-afa9a56afea9 -i td/osd-scrub-repair/1/new.json 2024-03-09T13:15:28.124 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:15:28.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:15:28.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvYOxlDz3yKhAArWPrpQnBZdTVoTztepIW0Q== --osd-uuid 55ce8a92-5cd3-49b9-823d-afa9a56afea9 2024-03-09T13:15:28.163 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:28.167+0000 7fdbcd4ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:15:28.283 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:28.287+0000 7fdbcd4ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:15:28.285 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:28.288+0000 7fdbcd4ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:15:28.285 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:28.289+0000 7fdbcd4ce640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:15:28.285 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:28.289+0000 7fdbcd4ce640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:15:28.285 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:28.289+0000 7fdbcd4ce640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:15:28.345 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:28.349+0000 7fdbcd4ce640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:15:34.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:15:34.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:15:34.059 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:15:34.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:15:34.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:15:34.415 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:15:34.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:15:34.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:15:34.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:15:34.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:15:34.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:15:34.443 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:34.444+0000 7ff46e415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:15:34.443 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:34.446+0000 7ff46e415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:15:34.444 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:34.447+0000 7ff46e415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:15:34.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:15:34.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:15:34.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:15:34.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:15:34.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:15:34.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:34.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:15:34.749 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:15:34.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:34.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:34.967 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:34.970+0000 7ff46e415640 -1 Falling back to public interface 2024-03-09T13:15:35.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:36.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:36.027 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:15:36.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:36.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:15:36.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:36.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:36.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:37.299 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:15:37.302+0000 7ff46e415640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:15:37.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:37.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:37.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:15:37.305 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:15:37.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:37.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:37.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:38.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:38.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:38.626 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:15:38.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:15:38.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:38.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:38.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:39.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:39.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:39.912 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:15:39.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:15:39.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:39.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:40.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:41.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:41.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:41.309 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:15:41.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:15:41.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:41.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:41.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:42.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:42.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:42.594 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:15:42.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:15:42.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:42.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:42.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:43.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:43.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:43.911 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:15:43.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:15:43.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:43.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:44.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:45.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:45.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:45.310 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:15:45.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:15:45.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:45.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:45.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:46.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:46.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:46.630 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:15:46.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:15:46.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:46.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:46.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:47.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:47.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:47.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:15:47.916 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:15:47.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:47.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:48.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:49.200 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:15:49.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:15:49.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:49.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:49.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:50.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:50.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:50.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:15:50.484 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:15:50.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:50.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:51.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:52.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:52.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:52.167 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:15:52.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:15:52.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:52.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:52.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:53.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:53.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:53.702 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:15:53.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:15:53.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:53.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:54.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:55.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:55.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:55.286 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:15:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:15:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:55.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:56.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:56.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:56.653 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:15:56.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:15:56.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:56.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:57.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:58.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:58.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:58.015 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:15:58.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:15:58.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:58.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:58.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:15:59.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:15:59.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:15:59.622 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:15:59.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:15:59.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:15:59.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:15:59.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:00.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:00.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:00.910 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:16:00.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:16:00.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:00.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:16:01.182 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/147761030,v1:127.0.0.1:6811/147761030] [v2:127.0.0.1:6812/147761030,v1:127.0.0.1:6813/147761030] exists,up 55ce8a92-5cd3-49b9-823d-afa9a56afea9 2024-03-09T13:16:01.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:16:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:16:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:16:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T13:16:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-03-09T13:16:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:16:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:16:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T13:16:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:16:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:16:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:16:01.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:16:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:16:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:16:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:16:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:16:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:16:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:16:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:16:01.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:16:01.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:16:01.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:16:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:16:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:16:01.188 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:16:01.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:16:01.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:16:01.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:16:01.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:16:01.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:16:01.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:16:01.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:16:01.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:16:01.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:16:01.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:16:01.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:16:01.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:16:01.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:16:01.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:16:01.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:16:01.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b6cd622-d0ea-48a0-93d3-25bde8cbd51a 2024-03-09T13:16:01.190 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 4b6cd622-d0ea-48a0-93d3-25bde8cbd51a 2024-03-09T13:16:01.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4b6cd622-d0ea-48a0-93d3-25bde8cbd51a' 2024-03-09T13:16:01.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:16:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARYexl56lQDBAAI+jzzX0GWd87OTAUwyTx1Q== 2024-03-09T13:16:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARYexl56lQDBAAI+jzzX0GWd87OTAUwyTx1Q=="}' 2024-03-09T13:16:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b6cd622-d0ea-48a0-93d3-25bde8cbd51a -i td/osd-scrub-repair/2/new.json 2024-03-09T13:16:01.544 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:16:01.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T13:16:01.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQARYexl56lQDBAAI+jzzX0GWd87OTAUwyTx1Q== --osd-uuid 4b6cd622-d0ea-48a0-93d3-25bde8cbd51a 2024-03-09T13:16:01.586 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:01.587+0000 7f58eb410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:16:01.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:01.632+0000 7f58eb410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:16:01.630 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:01.633+0000 7f58eb410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:16:01.630 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:01.634+0000 7f58eb410640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:16:01.630 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:01.634+0000 7f58eb410640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:16:01.630 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:01.634+0000 7f58eb410640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:16:01.699 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:01.703+0000 7f58eb410640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T13:16:07.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T13:16:07.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:16:07.436 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T13:16:07.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T13:16:07.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:16:07.790 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T13:16:07.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T13:16:07.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:16:07.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:16:07.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:16:07.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:16:07.817 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:07.820+0000 7fafa4417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:16:07.819 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:07.822+0000 7fafa4417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:16:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:07.823+0000 7fafa4417640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:16:08.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T13:16:08.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:16:08.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:16:08.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:16:08.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:16:08.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:08.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:16:08.064 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:16:08.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:08.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:08.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:08.355 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:08.358+0000 7fafa4417640 -1 Falling back to public interface 2024-03-09T13:16:09.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:09.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:09.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:16:09.336 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:16:09.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:09.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:09.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:10.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:10.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:10.634 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:16:10.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:16:10.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:10.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:10.752 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:10.756+0000 7fafa4417640 -1 osd.2 0 log_to_monitors true 2024-03-09T13:16:10.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:11.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:11.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:11.936 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:16:11.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:16:11.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:11.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:12.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:13.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:13.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:13.220 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:16:13.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:16:13.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:13.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:13.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:14.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:14.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:14.608 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:16:14.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:16:14.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:14.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:14.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:15.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:15.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:15.958 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:16:15.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:16:15.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:15.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:16.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:17.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:17.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:17.241 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:16:17.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:16:17.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:17.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:17.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:18.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:18.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:18.616 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:16:18.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:16:18.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:18.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:18.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:19.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:19.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:19.946 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:16:19.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:16:19.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:19.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:20.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:21.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:21.231 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:16:21.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:21.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:16:21.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:21.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:21.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:22.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:22.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:22.607 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:16:22.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:16:22.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:22.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:23.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:24.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:24.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:24.092 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:16:24.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:16:24.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:24.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:24.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:25.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:25.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:25.804 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:16:25.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:16:25.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:25.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:26.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:27.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:27.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:27.515 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:16:27.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:16:27.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:27.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:28.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:29.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:29.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:29.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:16:29.184 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:16:29.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:29.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:29.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:30.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:30.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:30.895 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:16:30.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:16:30.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:30.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:31.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:32.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:32.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:32.561 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:16:32.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:16:32.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:32.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:33.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:34.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:34.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:34.443 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:16:34.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:16:34.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:34.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:16:34.721 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3860755795,v1:127.0.0.1:6819/3860755795] [v2:127.0.0.1:6820/3860755795,v1:127.0.0.1:6821/3860755795] exists,up 4b6cd622-d0ea-48a0-93d3-25bde8cbd51a 2024-03-09T13:16:34.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:16:34.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:16:34.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:16:34.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T13:16:34.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-03-09T13:16:34.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:16:34.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:16:34.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-09T13:16:34.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:16:34.726 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:16:34.727 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:16:34.727 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:16:34.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:16:34.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:16:34.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:16:34.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:16:34.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:16:34.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:16:34.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:16:34.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:16:34.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:16:34.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:16:34.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:16:34.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:16:34.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:16:34.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:16:34.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:16:34.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d1331742-688f-482e-bfcf-9c00df7cdfa3 2024-03-09T13:16:34.728 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 d1331742-688f-482e-bfcf-9c00df7cdfa3 2024-03-09T13:16:34.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 d1331742-688f-482e-bfcf-9c00df7cdfa3' 2024-03-09T13:16:34.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:16:34.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAyYexlMKRqLBAAWwZfSWFweRIVV3S40m40aA== 2024-03-09T13:16:34.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAyYexlMKRqLBAAWwZfSWFweRIVV3S40m40aA=="}' 2024-03-09T13:16:34.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d1331742-688f-482e-bfcf-9c00df7cdfa3 -i td/osd-scrub-repair/3/new.json 2024-03-09T13:16:35.085 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:16:35.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-09T13:16:35.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAyYexlMKRqLBAAWwZfSWFweRIVV3S40m40aA== --osd-uuid d1331742-688f-482e-bfcf-9c00df7cdfa3 2024-03-09T13:16:35.124 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:35.127+0000 7f72dae01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:16:35.170 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:35.172+0000 7f72dae01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:16:35.170 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:35.173+0000 7f72dae01640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:16:35.171 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:35.173+0000 7f72dae01640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T13:16:35.171 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:35.173+0000 7f72dae01640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T13:16:35.171 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:35.173+0000 7f72dae01640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T13:16:35.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:35.263+0000 7f72dae01640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-09T13:16:41.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-09T13:16:41.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:16:41.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-09T13:16:41.263 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-03-09T13:16:41.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:16:41.654 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-03-09T13:16:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-09T13:16:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:16:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:16:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:16:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:16:41.681 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:41.683+0000 7f305e011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:16:41.682 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:41.685+0000 7f305e011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:16:41.683 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:41.686+0000 7f305e011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:16:41.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-09T13:16:41.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:16:41.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:16:41.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:16:41.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:16:41.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:41.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:16:41.931 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:16:41.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:41.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:42.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:42.202 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:42.205+0000 7f305e011640 -1 Falling back to public interface 2024-03-09T13:16:43.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:43.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:43.203 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:16:43.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:16:43.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:43.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:43.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:44.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:44.485 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:16:44.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:44.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:16:44.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:44.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:44.525 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:16:44.527+0000 7f305e011640 -1 osd.3 0 log_to_monitors true 2024-03-09T13:16:44.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:45.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:45.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:45.761 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:16:45.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:16:45.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:45.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:46.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:47.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:47.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:47.063 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:16:47.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:16:47.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:47.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:48.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:48.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:48.349 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:16:48.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:16:48.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:48.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:48.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:49.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:49.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:49.669 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:16:49.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:16:49.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:49.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:50.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:51.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:51.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:51.030 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:16:51.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:16:51.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:51.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:51.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:52.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:52.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:52.315 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:16:52.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:16:52.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:52.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:52.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:53.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:53.599 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:16:53.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:53.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:16:53.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:53.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:53.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:54.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:54.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:54.884 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:16:54.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:16:54.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:54.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:55.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:56.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:56.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:56.170 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:16:56.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:16:56.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:56.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:56.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:57.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:57.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:57.772 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:16:57.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:16:57.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:57.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:58.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:16:59.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:16:59.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:16:59.306 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:16:59.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:16:59.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:16:59.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:16:59.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:00.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:00.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:00.631 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:17:00.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:17:00.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:00.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:17:01.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:02.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:02.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:02.327 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:17:02.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:17:02.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:02.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:17:02.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:03.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:03.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:03.952 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:17:03.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:17:03.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:03.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:17:04.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:05.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:05.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:05.430 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:17:05.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:17:05.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:05.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:17:06.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:07.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:07.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:07.051 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:17:07.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:17:07.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:07.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:17:07.335 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1135818665,v1:127.0.0.1:6827/1135818665] [v2:127.0.0.1:6828/1135818665,v1:127.0.0.1:6829/1135818665] exists,up d1331742-688f-482e-bfcf-9c00df7cdfa3 2024-03-09T13:17:07.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:17:07.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:17:07.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:17:07.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-03-09T13:17:07.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T13:17:07.628 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T13:17:07.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T13:17:07.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T13:17:07.977 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T13:17:07.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:17:08.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T13:17:11.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-03-09T13:17:11.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:17:11.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:17:11.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:17:11.160 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:17:11.161 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:17:11.161 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:17:11.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:17:11.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:17:11.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:17:11.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:17:11.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:17:11.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:17:11.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:17:11.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:17:11.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:17:11.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:17:11.531 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:17:11.531 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:17:11.531 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-03-09T13:17:11.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:17:11.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:11.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:17:11.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-09T13:17:11.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-09T13:17:11.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-09T13:17:11.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:11.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:17:11.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-09T13:17:11.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-09T13:17:11.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672978' 2024-03-09T13:17:11.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:11.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:17:11.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509454 2024-03-09T13:17:11.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509454 2024-03-09T13:17:11.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672978 2-64424509454' 2024-03-09T13:17:11.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:11.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:17:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345923 2024-03-09T13:17:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345923 2024-03-09T13:17:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672978 2-64424509454 3-85899345923' 2024-03-09T13:17:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:11.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-09T13:17:11.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:17:11.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-09T13:17:11.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:11.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-09T13:17:11.977 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836503 2024-03-09T13:17:11.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-09T13:17:11.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:17:12.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-09T13:17:12.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:17:13.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:17:13.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:17:13.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-09T13:17:13.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:13.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-09T13:17:13.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:13.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:17:13.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-09T13:17:13.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:13.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-09T13:17:13.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-09T13:17:13.536 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672978 2024-03-09T13:17:13.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:17:13.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672978 2024-03-09T13:17:13.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:13.849 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509454 2024-03-09T13:17:13.849 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:13.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:17:13.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509454 2024-03-09T13:17:13.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:13.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509454 2024-03-09T13:17:13.851 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509454 2024-03-09T13:17:13.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-03-09T13:17:13.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:17:14.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509454 2024-03-09T13:17:14.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:14.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345923 2024-03-09T13:17:14.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:14.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:17:14.126 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345923 2024-03-09T13:17:14.126 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:14.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345923 2024-03-09T13:17:14.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345923' 2024-03-09T13:17:14.127 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345923 2024-03-09T13:17:14.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:17:14.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345924 -lt 85899345923 2024-03-09T13:17:14.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:17:14.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:17:14.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:17:14.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T13:17:14.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:17:14.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:17:14.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:17:14.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:17:14.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:17:14.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:17:14.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:17:15.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:17:15.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:17:15.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:17:15.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:17:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T13:17:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:17:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:17:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2024-03-09T13:17:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T13:17:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T13:17:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-09T13:17:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T13:17:15.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-03-09T13:17:15.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T13:17:15.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T13:17:16.441 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' created 2024-03-09T13:17:16.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:17:17.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-09T13:17:17.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T13:17:17.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:17:17.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:17:17.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:17:17.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:17:17.456 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:17:17.456 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:17:17.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:17:17.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:17:17.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:17:17.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:17:17.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:17:17.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:17:17.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:17:17.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:17:17.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:17:17.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:17:17.828 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:17:17.828 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:17:17.828 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-03-09T13:17:17.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:17:17.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:17.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:17:17.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-03-09T13:17:17.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-03-09T13:17:17.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-03-09T13:17:17.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:17.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:17:18.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-09T13:17:18.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-09T13:17:18.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672980' 2024-03-09T13:17:18.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:18.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:17:18.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-09T13:17:18.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-09T13:17:18.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672980 2-64424509456' 2024-03-09T13:17:18.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:18.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:17:18.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345926 2024-03-09T13:17:18.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345926 2024-03-09T13:17:18.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-42949672980 2-64424509456 3-85899345926' 2024-03-09T13:17:18.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:18.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-03-09T13:17:18.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:18.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:17:18.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-03-09T13:17:18.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:18.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-03-09T13:17:18.350 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836506 2024-03-09T13:17:18.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-03-09T13:17:18.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:17:18.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836506 2024-03-09T13:17:18.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:17:19.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:17:19.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:17:19.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-03-09T13:17:19.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:19.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-09T13:17:19.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:19.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:17:19.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-09T13:17:19.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:19.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-09T13:17:19.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-09T13:17:19.926 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672980 2024-03-09T13:17:19.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:17:20.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-03-09T13:17:20.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:20.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-09T13:17:20.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:20.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:17:20.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-09T13:17:20.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:20.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-09T13:17:20.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-09T13:17:20.204 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509456 2024-03-09T13:17:20.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:17:20.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-03-09T13:17:20.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:20.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345926 2024-03-09T13:17:20.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:20.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:17:20.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345926 2024-03-09T13:17:20.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:20.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345926 2024-03-09T13:17:20.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-03-09T13:17:20.480 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345926 2024-03-09T13:17:20.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:17:20.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345926 -lt 85899345926 2024-03-09T13:17:20.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:17:20.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:17:20.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:17:21.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:17:21.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:17:21.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:17:21.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:17:21.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:17:21.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:17:21.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:17:21.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:17:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:17:21.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:17:21.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:17:21.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:17:21.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:17:21.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:17:21.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:17:21.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T13:17:21.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-09T13:17:21.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T13:17:21.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-09T13:17:21.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-09T13:17:21.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:17:21.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:17:21.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-09T13:17:21.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:17:21.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:17:21.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:17:22.144 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:17:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:17:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:17:22.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:17:22.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:17:22.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T13:17:23.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-09T13:17:23.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T13:17:23.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:17:23.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:17:23.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:17:23.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-09T13:17:23.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-09T13:17:23.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T13:17:23.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-09T13:17:23.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-09T13:17:23.278 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:17:23.278 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T13:17:23.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-09T13:17:23.554 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:17:23.554 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:17:23.554 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T13:17:23.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-03-09T13:17:23.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-03-09T13:17:23.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-09T13:17:23.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-03-09T13:17:23.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-03-09T13:17:23.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-09T13:17:23.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T13:17:23.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T13:17:23.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-09T13:17:23.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:17:23.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:17:23.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:17:23.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:17:23.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:17:23.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:17:23.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:17:23.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:17:23.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:17:23.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:17:23.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:17:23.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:17:23.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:17:23.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:17:23.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:17:23.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:17:23.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:17:23.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:17:23.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:17:23.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:17:23.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:17:23.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:17:23.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:17:23.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T13:17:25.346 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T13:17:25.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:17:25.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:17:25.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:17:25.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:17:25.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:17:25.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:17:25.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:17:25.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:17:25.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:17:25.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:17:25.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:17:25.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:17:25.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:17:25.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:17:25.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:17:25.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:17:25.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:17:25.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:17:25.871 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:17:25.871 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:17:25.871 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:17:25.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:17:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:17:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:17:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:17:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:17:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:17:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:17:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:17:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:17:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:17:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:17:25.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:17:25.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:17:25.874 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-03-09T13:17:25.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:17:25.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:17:25.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:17:25.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:17:25.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:17:25.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:17:25.900 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:25.902+0000 7f748f80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:17:25.902 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:25.904+0000 7f748f80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:17:25.904 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:25.905+0000 7f748f80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:17:26.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:17:26.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:17:26.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:17:26.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:17:26.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:17:26.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:26.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:17:26.162 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:17:26.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:26.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:17:26.424 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:26.426+0000 7f748f80c640 -1 Falling back to public interface 2024-03-09T13:17:26.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:27.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:27.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:27.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:17:27.441 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:17:27.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:27.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:17:27.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:28.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:28.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:28.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:17:28.728 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:17:28.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:28.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:17:28.818 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:28.819+0000 7f748f80c640 -1 osd.3 36 log_to_monitors true 2024-03-09T13:17:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:30.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:30.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:30.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:17:30.015 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:17:30.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:30.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:17:30.351 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/360286867,v1:127.0.0.1:6827/360286867] [v2:127.0.0.1:6828/360286867,v1:127.0.0.1:6829/360286867] exists,up d1331742-688f-482e-bfcf-9c00df7cdfa3 2024-03-09T13:17:30.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:17:30.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:17:30.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:17:30.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:17:30.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:17:30.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:17:30.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:17:30.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:17:30.351 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:17:30.351 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:17:30.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:17:30.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:17:30.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:17:30.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:17:30.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:17:30.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:17:30.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:17:30.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:17:30.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:17:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:17:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:17:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:17:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-03-09T13:17:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:17:30.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:30.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:17:30.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836509 2024-03-09T13:17:30.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836509 2024-03-09T13:17:30.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509' 2024-03-09T13:17:30.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:30.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:17:31.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672984 2024-03-09T13:17:31.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672984 2024-03-09T13:17:31.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672984' 2024-03-09T13:17:31.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:31.102 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:17:31.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-03-09T13:17:31.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-03-09T13:17:31.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672984 2-64424509460' 2024-03-09T13:17:31.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:31.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:17:31.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-09T13:17:31.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-09T13:17:31.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836509 1-42949672984 2-64424509460 3-171798691842' 2024-03-09T13:17:31.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:31.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836509 2024-03-09T13:17:31.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:31.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:17:31.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836509 2024-03-09T13:17:31.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:31.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836509 2024-03-09T13:17:31.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-03-09T13:17:31.311 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836509 2024-03-09T13:17:31.312 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:17:31.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836509 2024-03-09T13:17:31.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:17:32.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:17:32.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:17:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836509 -lt 21474836509 2024-03-09T13:17:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:32.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672984 2024-03-09T13:17:32.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:17:32.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672984 2024-03-09T13:17:32.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672984 2024-03-09T13:17:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-03-09T13:17:32.899 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672984 2024-03-09T13:17:32.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:17:33.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672984 -lt 42949672984 2024-03-09T13:17:33.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:33.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-03-09T13:17:33.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:33.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:17:33.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-03-09T13:17:33.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:33.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-03-09T13:17:33.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-03-09T13:17:33.174 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509460 2024-03-09T13:17:33.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:17:33.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-03-09T13:17:33.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:33.447 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-171798691842 2024-03-09T13:17:33.447 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:33.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:17:33.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-171798691842 2024-03-09T13:17:33.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:33.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-09T13:17:33.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 171798691842' 2024-03-09T13:17:33.449 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 171798691842 2024-03-09T13:17:33.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:17:33.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-09T13:17:33.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:17:33.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:17:33.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:17:34.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:17:34.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:17:34.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:17:34.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:17:34.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:17:34.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:17:34.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:17:34.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:17:34.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:17:34.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:17:34.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:17:34.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:17:34.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:17:34.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:17:34.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:17:34.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T13:17:34.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:17:34.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:17:34.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:17:34.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:17:35.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T13:17:35.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T13:17:35.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:17:35.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:17:35.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:17:35.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:17:35.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:17:35.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:17:35.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:17:16.335262+0000 2024-03-09T13:17:35.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:17:35.550 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:17:35.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:17:16.335262+0000 2024-03-09T13:17:35.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:17:35.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:17:16.335262+0000 2024-03-09T13:17:35.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:17:35.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:17:35.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:17:35.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:17:35.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:17:35.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:17:35.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:17:35.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:17:35.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:17:16.335262+0000 '>' 2024-03-09T13:17:16.335262+0000 2024-03-09T13:17:35.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:17:36.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:17:36.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:17:36.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:17:36.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:17:36.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:17:36.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:17:36.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:17:37.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:17:16.335262+0000 '>' 2024-03-09T13:17:16.335262+0000 2024-03-09T13:17:37.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:17:38.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:17:38.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:17:38.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:17:38.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:17:38.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:17:38.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:17:38.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:17:38.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:17:16.335262+0000 '>' 2024-03-09T13:17:16.335262+0000 2024-03-09T13:17:38.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:17:39.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:17:39.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:17:39.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:17:39.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:17:39.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:17:39.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:17:39.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:17:39.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:17:16.335262+0000 '>' 2024-03-09T13:17:16.335262+0000 2024-03-09T13:17:39.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:17:40.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:17:40.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:17:40.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:17:40.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:17:40.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:17:40.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:17:40.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:17:40.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:17:35.644315+0000 '>' 2024-03-09T13:17:16.335262+0000 2024-03-09T13:17:40.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:17:40.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:17:40.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:17:40.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:17:41.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:17:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:17:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:17:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:17:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:17:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:17:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:17:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T13:17:42.576 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T13:17:42.577 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:17:42.577 INFO:tasks.workunit.client.0.smithi162.stdout:hinfo_key 2024-03-09T13:17:42.577 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:17:42.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:17:42.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:17:42.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:17:42.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:17:42.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:17:42.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:17:42.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:17:42.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:17:42.846 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-03-09T13:17:42.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:17:42.847 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:17:42.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:17:42.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:17:42.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:17:42.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:17:42.873 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:42.875+0000 7f491f0c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:17:42.875 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:42.878+0000 7f491f0c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:17:42.876 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:42.879+0000 7f491f0c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:17:43.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:17:43.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:17:43.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:17:43.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:17:43.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:17:43.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:43.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:17:43.129 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:17:43.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:43.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:17:43.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:43.403 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:43.406+0000 7f491f0c6640 -1 Falling back to public interface 2024-03-09T13:17:44.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:44.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:44.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:17:44.400 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:17:44.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:44.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:17:44.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:45.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:45.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:45.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:17:45.685 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:17:45.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:45.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:17:45.758 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:45.761+0000 7f491f0c6640 -1 osd.3 42 log_to_monitors true 2024-03-09T13:17:45.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:46.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:46.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:46.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:17:46.971 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:17:46.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:46.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:17:47.257 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 46 up_thru 46 down_at 43 last_clean_interval [40,42) [v2:127.0.0.1:6826/2818450918,v1:127.0.0.1:6827/2818450918] [v2:127.0.0.1:6828/2818450918,v1:127.0.0.1:6829/2818450918] exists,up d1331742-688f-482e-bfcf-9c00df7cdfa3 2024-03-09T13:17:47.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:17:47.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:17:47.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:17:47.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:17:47.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:17:47.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:17:47.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:17:47.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:17:47.258 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:17:47.258 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:17:47.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:17:47.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:17:47.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:17:47.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:17:47.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:17:47.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:17:47.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:17:47.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:17:47.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:17:47.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:17:47.657 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:17:47.657 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:17:47.657 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-03-09T13:17:47.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:17:47.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:47.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:17:47.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836513 2024-03-09T13:17:47.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836513 2024-03-09T13:17:47.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513' 2024-03-09T13:17:47.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:47.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:17:47.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672988 2024-03-09T13:17:47.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672988 2024-03-09T13:17:47.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672988' 2024-03-09T13:17:47.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:47.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:17:47.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-03-09T13:17:47.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-03-09T13:17:47.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672988 2-64424509464' 2024-03-09T13:17:47.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:47.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:17:48.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495618 2024-03-09T13:17:48.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495618 2024-03-09T13:17:48.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836513 1-42949672988 2-64424509464 3-197568495618' 2024-03-09T13:17:48.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:48.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836513 2024-03-09T13:17:48.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:48.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:17:48.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836513 2024-03-09T13:17:48.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:48.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836513 2024-03-09T13:17:48.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-03-09T13:17:48.076 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836513 2024-03-09T13:17:48.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:17:48.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836513 2024-03-09T13:17:48.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:17:49.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:17:49.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:17:49.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836513 2024-03-09T13:17:49.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:49.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672988 2024-03-09T13:17:49.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:49.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:17:49.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672988 2024-03-09T13:17:49.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:49.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672988 2024-03-09T13:17:49.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-03-09T13:17:49.663 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672988 2024-03-09T13:17:49.663 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:17:49.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672988 -lt 42949672988 2024-03-09T13:17:49.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:49.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-03-09T13:17:49.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:49.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:17:49.942 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-03-09T13:17:49.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:49.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-03-09T13:17:49.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-03-09T13:17:49.943 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509464 2024-03-09T13:17:49.944 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:17:50.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-03-09T13:17:50.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:50.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495618 2024-03-09T13:17:50.216 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:50.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:17:50.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495618 2024-03-09T13:17:50.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:50.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495618 2024-03-09T13:17:50.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495618' 2024-03-09T13:17:50.218 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 197568495618 2024-03-09T13:17:50.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:17:50.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495618 -lt 197568495618 2024-03-09T13:17:50.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:17:50.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:17:50.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:17:50.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:17:50.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:17:50.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:17:50.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:17:50.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:17:50.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:17:50.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:17:50.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:17:51.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:17:51.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:17:51.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:17:51.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:17:51.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:17:51.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:17:51.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:17:51.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:17:51.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:17:51.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-03-09T13:17:51.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T13:17:51.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T13:17:51.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-09T13:17:51.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:17:51.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:17:51.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:17:51.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:17:51.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:17:51.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:17:51.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:17:51.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:17:51.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:17:51.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:17:51.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:17:51.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:17:51.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:17:51.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:17:51.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:17:51.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:17:51.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:17:51.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:17:51.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:17:51.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:17:51.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:17:51.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:17:51.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:17:51.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T13:17:53.714 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:17:54.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:17:54.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:17:54.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:17:54.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:17:54.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:17:54.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:17:54.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:17:54.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:17:54.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:17:54.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:17:54.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:17:54.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:17:54.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:17:54.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:17:54.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:17:54.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:17:54.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:17:54.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:17:54.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:17:54.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:17:54.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:17:54.234 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:17:54.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:17:54.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:17:54.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:17:54.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:17:54.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:17:54.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:17:54.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:54.264+0000 7f81b4a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:17:54.264 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:54.266+0000 7f81b4a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:17:54.265 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:54.268+0000 7f81b4a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:17:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:17:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:17:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:17:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:17:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:17:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:17:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:17:54.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:54.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:17:54.780 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:54.783+0000 7f81b4a13640 -1 Falling back to public interface 2024-03-09T13:17:54.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:55.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:55.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:55.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:17:55.820 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:17:55.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:55.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:17:56.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:57.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:57.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:57.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:17:57.095 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:17:57.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:57.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:17:57.154 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:17:57.155+0000 7f81b4a13640 -1 osd.1 48 log_to_monitors true 2024-03-09T13:17:57.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:17:58.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:17:58.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:17:58.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:17:58.373 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:17:58.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:17:58.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:17:58.682 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 53 up_thru 53 down_at 49 last_clean_interval [10,48) [v2:127.0.0.1:6810/3183609546,v1:127.0.0.1:6811/3183609546] [v2:127.0.0.1:6812/3183609546,v1:127.0.0.1:6813/3183609546] exists,up 55ce8a92-5cd3-49b9-823d-afa9a56afea9 2024-03-09T13:17:58.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:17:58.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:17:58.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:17:58.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:17:58.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:17:58.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:17:58.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:17:58.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:17:58.682 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:17:58.682 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:17:58.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:17:58.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:17:58.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:17:58.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:17:58.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:17:58.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:17:58.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:17:58.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:17:58.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:17:59.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:17:59.095 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:17:59.095 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:17:59.095 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-03-09T13:17:59.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:17:59.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:59.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:17:59.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836517 2024-03-09T13:17:59.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836517 2024-03-09T13:17:59.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517' 2024-03-09T13:17:59.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:59.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:17:59.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266690 2024-03-09T13:17:59.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266690 2024-03-09T13:17:59.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-227633266690' 2024-03-09T13:17:59.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:59.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:17:59.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509468 2024-03-09T13:17:59.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509468 2024-03-09T13:17:59.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-227633266690 2-64424509468' 2024-03-09T13:17:59.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:17:59.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:17:59.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=197568495621 2024-03-09T13:17:59.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 197568495621 2024-03-09T13:17:59.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836517 1-227633266690 2-64424509468 3-197568495621' 2024-03-09T13:17:59.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:17:59.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836517 2024-03-09T13:17:59.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:17:59.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:17:59.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836517 2024-03-09T13:17:59.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:17:59.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836517 2024-03-09T13:17:59.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-03-09T13:17:59.542 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836517 2024-03-09T13:17:59.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:17:59.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836517 2024-03-09T13:17:59.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:18:00.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:18:00.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:01.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836517 -lt 21474836517 2024-03-09T13:18:01.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:01.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266690 2024-03-09T13:18:01.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:01.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:18:01.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266690 2024-03-09T13:18:01.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:01.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266690 2024-03-09T13:18:01.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266690' 2024-03-09T13:18:01.122 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 227633266690 2024-03-09T13:18:01.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:18:01.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266690 -lt 227633266690 2024-03-09T13:18:01.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:01.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509468 2024-03-09T13:18:01.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:01.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:18:01.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509468 2024-03-09T13:18:01.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:01.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509468 2024-03-09T13:18:01.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-03-09T13:18:01.397 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509468 2024-03-09T13:18:01.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:18:01.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509468 -lt 64424509468 2024-03-09T13:18:01.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:01.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-197568495621 2024-03-09T13:18:01.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:01.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:18:01.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-197568495621 2024-03-09T13:18:01.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:01.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=197568495621 2024-03-09T13:18:01.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 197568495621' 2024-03-09T13:18:01.700 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 197568495621 2024-03-09T13:18:01.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:18:01.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 197568495621 -lt 197568495621 2024-03-09T13:18:01.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:18:01.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:01.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:02.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:18:02.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:18:02.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:18:02.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:18:02.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:18:02.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:18:02.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:18:02.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:18:02.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:18:02.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:18:02.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:02.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:02.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:18:02.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:18:02.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:18:02.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T13:18:02.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:18:02.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:18:02.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:18:02.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:18:03.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T13:18:03.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T13:18:03.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:18:03.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:18:03.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:18:03.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:18:03.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:18:03.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:18:03.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:17:35.644315+0000 2024-03-09T13:18:03.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:18:03.771 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:18:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:17:35.644315+0000 2024-03-09T13:18:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:18:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:17:35.644315+0000 2024-03-09T13:18:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:18:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:18:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:18:03.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:18:03.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:18:03.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:18:03.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:18:03.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:18:04.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:17:35.644315+0000 '>' 2024-03-09T13:17:35.644315+0000 2024-03-09T13:18:04.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:18:05.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:18:05.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:18:05.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:18:05.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:18:05.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:18:05.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:18:05.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:18:05.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:17:35.644315+0000 '>' 2024-03-09T13:17:35.644315+0000 2024-03-09T13:18:05.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:18:06.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:18:06.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:18:06.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:18:06.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:18:06.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:18:06.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:18:06.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:18:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:18:04.712880+0000 '>' 2024-03-09T13:17:35.644315+0000 2024-03-09T13:18:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:18:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:18:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:18:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:18:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:18:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:18:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:18:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:18:06.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:18:06.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:18:06.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:18:06.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:18:06.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:18:06.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:18:06.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:18:06.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:18:06.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:18:06.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:18:06.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:18:06.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:18:06.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:18:06.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:18:06.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:18:06.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:18:06.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T13:18:08.237 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T13:18:08.237 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:18:08.237 INFO:tasks.workunit.client.0.smithi162.stdout:hinfo_key 2024-03-09T13:18:08.237 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:18:08.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:18:08.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:18:08.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:18:08.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:18:08.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:18:08.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:18:08.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:18:08.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:18:08.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:18:08.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:18:08.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:18:08.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:18:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:18:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:18:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:18:08.506 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:18:08.506 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:18:08.506 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:18:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:18:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:18:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:18:08.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:08.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:08.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:18:08.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:18:08.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:18:08.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:18:08.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:18:08.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:18:08.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:18:08.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:18:08.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:18:08.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:18:08.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:18:08.508 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-03-09T13:18:08.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:18:08.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:18:08.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:18:08.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:18:08.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:18:08.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:18:08.534 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:18:08.535+0000 7f88145aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:08.536 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:18:08.538+0000 7f88145aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:08.537 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:18:08.539+0000 7f88145aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:08.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:18:08.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:18:08.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:18:08.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:18:08.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:18:08.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:08.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:18:08.792 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:18:08.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:08.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:18:09.051 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:18:09.053+0000 7f88145aa640 -1 Falling back to public interface 2024-03-09T13:18:09.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:10.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:10.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:10.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:18:10.066 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:18:10.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:10.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:18:10.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:11.228 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:18:11.230+0000 7f88145aa640 -1 osd.3 55 log_to_monitors true 2024-03-09T13:18:11.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:11.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:11.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:18:11.346 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:18:11.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:11.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:18:11.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:12.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:12.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:12.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:18:12.626 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:18:12.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:12.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:18:12.909 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 59 up_thru 59 down_at 56 last_clean_interval [46,55) [v2:127.0.0.1:6826/2072165078,v1:127.0.0.1:6827/2072165078] [v2:127.0.0.1:6828/2072165078,v1:127.0.0.1:6829/2072165078] exists,up d1331742-688f-482e-bfcf-9c00df7cdfa3 2024-03-09T13:18:12.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:18:12.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:18:12.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:18:12.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:18:12.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:18:12.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:18:12.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:18:12.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:18:12.910 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:18:12.910 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:18:12.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:18:12.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:18:12.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:18:12.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:18:12.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:18:12.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:18:12.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:18:12.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:18:12.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:18:13.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:18:13.270 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:18:13.270 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:18:13.270 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-03-09T13:18:13.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:18:13.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:13.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:18:13.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836521 2024-03-09T13:18:13.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836521 2024-03-09T13:18:13.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521' 2024-03-09T13:18:13.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:13.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:18:13.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266694 2024-03-09T13:18:13.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266694 2024-03-09T13:18:13.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-227633266694' 2024-03-09T13:18:13.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:13.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:18:13.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509471 2024-03-09T13:18:13.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509471 2024-03-09T13:18:13.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-227633266694 2-64424509471' 2024-03-09T13:18:13.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:13.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:18:13.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070466 2024-03-09T13:18:13.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070466 2024-03-09T13:18:13.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836521 1-227633266694 2-64424509471 3-253403070466' 2024-03-09T13:18:13.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:13.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836521 2024-03-09T13:18:13.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:13.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:18:13.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836521 2024-03-09T13:18:13.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:13.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836521 2024-03-09T13:18:13.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-03-09T13:18:13.723 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836521 2024-03-09T13:18:13.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:13.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836519 -lt 21474836521 2024-03-09T13:18:13.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:18:14.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:18:14.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:15.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836521 -lt 21474836521 2024-03-09T13:18:15.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:15.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-227633266694 2024-03-09T13:18:15.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:15.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:18:15.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-227633266694 2024-03-09T13:18:15.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:15.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266694 2024-03-09T13:18:15.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 227633266694' 2024-03-09T13:18:15.281 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 227633266694 2024-03-09T13:18:15.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:18:15.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266694 -lt 227633266694 2024-03-09T13:18:15.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:15.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509471 2024-03-09T13:18:15.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:15.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:18:15.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509471 2024-03-09T13:18:15.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:15.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509471 2024-03-09T13:18:15.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-03-09T13:18:15.561 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509471 2024-03-09T13:18:15.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:18:15.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509472 -lt 64424509471 2024-03-09T13:18:15.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:15.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070466 2024-03-09T13:18:15.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:15.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:18:15.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070466 2024-03-09T13:18:15.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:15.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070466 2024-03-09T13:18:15.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070466' 2024-03-09T13:18:15.880 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 253403070466 2024-03-09T13:18:15.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:18:16.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070466 -lt 253403070466 2024-03-09T13:18:16.156 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:18:16.156 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:16.156 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:16.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:18:16.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:18:16.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:18:16.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:18:16.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:18:16.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:18:16.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:18:16.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:18:16.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:18:16.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:18:16.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:16.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:17.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:18:17.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:18:17.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:18:17.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:18:17.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:18:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-03-09T13:18:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T13:18:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T13:18:17.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165924"' 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165924' 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 165926"' 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 165926' 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T13:18:17.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 165924 165926' 2024-03-09T13:18:17.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165927: /' 2024-03-09T13:18:17.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:18:17.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:18:17.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165924 2024-03-09T13:18:17.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T13:18:17.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/165929: /' 2024-03-09T13:18:19.957 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:18:19.957 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:18:19.957 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:18:19.957 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:18:19.957 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:18:19.957 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:18:19.958 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:18:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:18:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:18:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T13:18:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:165927: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:18:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:18:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:18:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:18:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:18:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:18:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:18:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:18:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:18:19.960 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:18:19.960 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:18:19.960 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:18:19.960 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:18:19.961 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:18:19.962 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:18:19.962 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:18:19.962 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:18:19.962 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:18:19.962 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:18:19.962 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:18:19.962 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:18:19.962 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:18:19.962 INFO:tasks.workunit.client.0.smithi162.stderr:165927: start osd.1 2024-03-09T13:18:19.962 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:18:20.014 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:18:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T13:18:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:18:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:18:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:18:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:18:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:18:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:18:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:18:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:18:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:18:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:18:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:18:20.018 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:18:20.019 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:18:20.019 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:18:20.019 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:18:20.019 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:18:20.019 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T13:18:20.019 INFO:tasks.workunit.client.0.smithi162.stderr:165929: start osd.2 2024-03-09T13:18:20.019 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:18:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:18:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:18:24.511 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:18:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:18:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:18:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:165927: 2024-03-09T13:18:19.990+0000 7fd195c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:165927: 2024-03-09T13:18:19.992+0000 7fd195c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:165927: 2024-03-09T13:18:19.993+0000 7fd195c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:18:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:18:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:18:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:18:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:18:24.512 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: 0 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: 2024-03-09T13:18:20.514+0000 7fd195c13640 -1 Falling back to public interface 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: 1 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:24.513 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:24.514 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:24.514 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:18:24.514 INFO:tasks.workunit.client.0.smithi162.stderr:165927: 2 2024-03-09T13:18:24.514 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:24.514 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:18:24.514 INFO:tasks.workunit.client.0.smithi162.stderr:165927: 2024-03-09T13:18:23.117+0000 7fd195c13640 -1 osd.1 60 log_to_monitors true 2024-03-09T13:18:24.514 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:24.514 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:24.514 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:24.514 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:18:24.514 INFO:tasks.workunit.client.0.smithi162.stderr:165927: 3 2024-03-09T13:18:24.514 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:24.514 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:18:24.779 INFO:tasks.workunit.client.0.smithi162.stderr:165927: osd.1 up in weight 1 up_from 65 up_thru 65 down_at 61 last_165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T13:18:24.780 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T13:18:24.780 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:18:24.780 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:18:24.780 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:18:24.780 INFO:tasks.workunit.client.0.smithi162.stderr:165929: 2024-03-09T13:18:20.045+0000 7f539f80b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:24.780 INFO:tasks.workunit.client.0.smithi162.stderr:165929: 2024-03-09T13:18:20.047+0000 7f539f80b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:24.780 INFO:tasks.workunit.client.0.smithi162.stderr:165929: 2024-03-09T13:18:20.049+0000 7f539f80b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:24.780 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T13:18:24.780 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:18:24.780 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:18:24.780 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:18:24.780 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:18:24.780 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: 0 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: 2024-03-09T13:18:20.578+0000 7f539f80b640 -1 Falling back to public interface 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: 1 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:24.781 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:18:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:165929: 2 2024-03-09T13:18:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:18:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:165929: 2024-03-09T13:18:23.421+0000 7f539f80b640 -1 osd.2 60 log_to_monitors true 2024-03-09T13:18:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:18:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:165929: 3 2024-03-09T13:18:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:24.782 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:18:25.255 INFO:tasks.workunit.client.0.smithi162.stderr:165929: osd.2 up in weight 1 up_from 66 up_thru 49 down_at 61 last_clean_interval [53,60) [v2:127.0.0.1:6810/2173730094,v1:127.0.0.1:6811/2173730094] [v2:127.0.0.1:6812/2173730094,v1:127.0.0.1:6813/2173730094] exists,up 55ce8a92-5cd3-49b9-823d-afa9a56afea9 2024-03-09T13:18:25.255 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:18:25.255 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:18:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:18:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:18:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:18:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:18:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:18:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:18:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:165927: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:18:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:165927: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:18:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:18:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:18:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:18:25.256 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:18:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:18:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:18:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:18:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:18:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:18:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:18:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:165927: 1 2024-03-09T13:18:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:165927: 2 2024-03-09T13:18:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:165927: 3' 2024-03-09T13:18:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:18:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:25.257 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:18:25.258 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836524 2024-03-09T13:18:25.258 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836524 2024-03-09T13:18:25.258 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524' 2024-03-09T13:18:25.258 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:25.258 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:18:25.258 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874242 2024-03-09T13:18:25.258 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874242 2024-03-09T13:18:25.258 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-279172874242' 2024-03-09T13:18:25.258 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:25.520 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [15,60) [v2:127.0.0.1:6818/1836358101,v1:127.0.0.1:6819/1836358101] [v2:127.0.0.1:6820/1836358101,v1:127.0.0.1:6821/1836358101] exists,up 4b6cd622-d0ea-48a0-93d3-25bde8cbd51a 2024-03-09T13:18:25.520 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:18:25.520 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:18:25.520 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:18:25.521 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:18:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:18:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:18:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:165929: 1 2024-03-09T13:18:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:165929: 2 2024-03-09T13:18:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:165929: 3' 2024-03-09T13:18:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:18:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:18:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836525 2024-03-09T13:18:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836525 2024-03-09T13:18:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525' 2024-03-09T13:18:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:25.522 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:18:25.523 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874243 2024-03-09T13:18:25.523 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874243 2024-03-09T13:18:25.523 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-279172874243' 2024-03-09T13:18:25.523 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:27.464 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:18:27.464 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-03-09T13:18:27.464 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-03-09T13:18:27.464 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-279172874242 2-283467841538' 2024-03-09T13:18:27.464 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:27.464 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:18:27.464 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070469 2024-03-09T13:18:27.464 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070469 2024-03-09T13:18:27.464 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836524 1-279172874242 2-283467841538 3-253403070469' 2024-03-09T13:18:27.464 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:27.464 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836524 2024-03-09T13:18:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:18:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836524 2024-03-09T13:18:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836524 2024-03-09T13:18:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-03-09T13:18:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:165927: waiting osd.0 seq 21474836524 2024-03-09T13:18:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836524 2024-03-09T13:18:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:18:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:18:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836524 2024-03-09T13:18:27.466 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:27.466 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874242 2024-03-09T13:18:27.466 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:27.466 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:18:27.466 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874242 2024-03-09T13:18:27.466 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:27.466 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874242 2024-03-09T13:18:27.466 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874242' 2024-03-09T13:18:27.466 INFO:tasks.workunit.client.0.smithi162.stderr:165927: waiting osd.1 seq 279172874242 2024-03-09T13:18:27.466 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:18:27.466 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874243 -lt 279172874242 2024-03-09T13:18:27.466 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:27.719 INFO:tasks.workunit.client.0.smithi162.stderr:1659tell osd.2 flush_pg_stats 2024-03-09T13:18:27.719 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841539 2024-03-09T13:18:27.719 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841539 2024-03-09T13:18:27.719 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-279172874243 2-283467841539' 2024-03-09T13:18:27.719 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:27.719 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:18:27.719 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070470 2024-03-09T13:18:27.719 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070470 2024-03-09T13:18:27.719 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836525 1-279172874243 2-283467841539 3-253403070470' 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836525 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836525 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836525 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: waiting osd.0 seq 21474836525 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836525 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:18:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836525 -lt 21474836525 2024-03-09T13:18:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-279172874243 2024-03-09T13:18:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:18:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-279172874243 2024-03-09T13:18:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874243 2024-03-09T13:18:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 279172874243' 2024-03-09T13:18:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:165929: waiting osd.1 seq 279172874243 2024-03-09T13:18:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:18:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874243 -lt 279172874243 2024-03-09T13:18:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:28.770 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-283467841538 2024-03-09T13:18:28.770 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:28.770 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:18:28.770 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-283467841538 2024-03-09T13:18:28.770 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:28.770 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-03-09T13:18:28.770 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 283467841538' 2024-03-09T13:18:28.770 INFO:tasks.workunit.client.0.smithi162.stderr:165927: waiting osd.2 seq 283467841538 2024-03-09T13:18:28.770 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:18:28.770 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841538 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070469 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070469 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070469 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070469' 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: waiting osd.3 seq 253403070469 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070471 -lt 253403070469 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:28.771 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:28.772 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:18:28.772 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:18:28.772 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:18:28.772 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:18:28.772 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:18:28.772 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:18:28.772 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:18:28.772 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:18:28.772 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:18:28.772 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:18:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.c29: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-283467841539 2024-03-09T13:18:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:18:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-283467841539 2024-03-09T13:18:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841539 2024-03-09T13:18:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 283467841539' 2024-03-09T13:18:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:165929: waiting osd.2 seq 283467841539 2024-03-09T13:18:29.012 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841539 -lt 283467841539 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070470 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070470 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070470 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070470' 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: waiting osd.3 seq 253403070470 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070471 -lt 253403070470 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:18:29.013 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:29.014 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:29.014 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:18:29.014 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:18:29.014 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:18:29.014 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:18:29.014 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:18:29.014 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:18:29.014 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:18:29.014 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:18:29.014 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:18:29.014 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:18:29.178 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:29.178 INFO:tasks.workunit.client.0.smithi162.stderr:165927: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:29.178 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:18:29.178 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:18:29.178 INFO:tasks.workunit.client.0.smithi162.stderr:165927: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:18:29.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:18:29.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:18:29.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 165926 2024-03-09T13:18:29.380 INFO:tasks.workunit.client.0.smithi162.stderr:lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:29.380 INFO:tasks.workunit.client.0.smithi162.stderr:165929: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:29.380 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:18:29.380 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:18:29.380 INFO:tasks.workunit.client.0.smithi162.stderr:165929: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:18:29.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:18:29.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:18:29.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:18:29.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:18:29.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T13:18:29.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:18:29.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T13:18:29.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:18:29.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:18:29.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:18:29.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:18:29.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T13:18:29.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T13:18:29.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:18:29.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:18:29.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:18:29.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:18:29.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:18:29.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:18:29.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:18:04.712880+0000 2024-03-09T13:18:29.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:18:30.192 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:18:30.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:18:04.712880+0000 2024-03-09T13:18:30.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:18:30.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:18:04.712880+0000 2024-03-09T13:18:30.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:18:30.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:18:30.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:18:30.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:18:30.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:18:30.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:18:30.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:18:30.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:18:30.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:18:04.712880+0000 '>' 2024-03-09T13:18:04.712880+0000 2024-03-09T13:18:30.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:18:31.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:18:31.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:18:31.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:18:31.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:18:31.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:18:31.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:18:31.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:18:31.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:18:04.712880+0000 '>' 2024-03-09T13:18:04.712880+0000 2024-03-09T13:18:31.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:18:32.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:18:32.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:18:32.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:18:32.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:18:32.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:18:32.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:18:32.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:18:33.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:18:31.073525+0000 '>' 2024-03-09T13:18:04.712880+0000 2024-03-09T13:18:33.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:18:33.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T13:18:33.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:18:33.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:18:33.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:18:33.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 168726"' 2024-03-09T13:18:33.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 168726' 2024-03-09T13:18:33.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T13:18:33.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:18:33.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:18:33.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 168728"' 2024-03-09T13:18:33.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:18:33.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 168728' 2024-03-09T13:18:33.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T13:18:33.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 168726 168728' 2024-03-09T13:18:33.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:18:33.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/168729: /' 2024-03-09T13:18:33.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:18:33.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 168726 2024-03-09T13:18:33.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T13:18:33.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/168731: /' 2024-03-09T13:18:35.663 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:18:35.663 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:18:35.663 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T13:18:35.663 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T13:18:35.664 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: _ 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: hinfo_key 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: snapset 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:18:35.665 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:18:35.666 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:18:35.666 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:18:35.666 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:18:35.666 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:18:35.666 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:18:35.666 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:18:35.666 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:18:35.667 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:18:35.668 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:18:35.668 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:18:35.668 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:18:35.668 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:18:35.668 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:18:35.668 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T13:18:35.668 INFO:tasks.workunit.client.0.smithi162.stderr:168731: start osd.2 2024-03-09T13:18:35.707 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:18:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:18:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:18:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:18:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:18:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:18:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:18:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:18:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:18:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:18:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:18:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:18:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:18:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:18:35.709 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:18:35.709 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:18:35.709 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:18:35.709 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:18:35.709 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:18:35.709 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:18:35.709 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:18:35.709 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:18:35.709 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T13:18:35.709 INFO:tasks.workunit.client.0.smithi162.stderr:168729: _ 2024-03-09T13:18:35.709 INFO:tasks.workunit.client.0.smithi162.stderr:168729: hinfo_key 2024-03-09T13:18:35.709 INFO:tasks.workunit.client.0.smithi162.stderr:168729: snapset 2024-03-09T13:18:35.709 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:18:35.710 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:18:35.710 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:18:35.710 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:18:35.710 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:18:35.710 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:18:35.710 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:18:35.710 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:18:35.710 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:18:35.710 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:18:35.710 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:18:35.711 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:18:35.711 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:18:35.711 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:18:35.711 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:18:35.711 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:18:35.711 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:18:35.711 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:18:35.711 INFO:tasks.workunit.client.0.smithi162.stderr:168729: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:18:35.711 INFO:tasks.workunit.client.0.smithi162.stderr:168729: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:18:35.711 INFO:tasks.workunit.client.0.smithi162.stderr:168729: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:18:35.711 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:35.711 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:35.711 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:18:35.712 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:18:35.712 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:18:35.712 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:18:35.712 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:18:35.712 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:18:35.712 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:18:35.712 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:18:35.712 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:18:35.712 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:18:35.712 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:18:35.712 INFO:tasks.workunit.client.0.smithi162.stderr:168729: start osd.1 2024-03-09T13:18:40.344 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:18:40.344 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:18:40.344 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:18:40.344 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:18:40.344 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:18:40.344 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:18:40.344 INFO:tasks.workunit.client.0.smithi162.stderr:168729: 2024-03-09T13:18:35.740+0000 7fa3c9ed3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:40.344 INFO:tasks.workunit.client.0.smithi162.stderr:168729: 2024-03-09T13:18:35.742+0000 7fa3c9ed3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:40.344 INFO:tasks.workunit.client.0.smithi162.stderr:168729: 2024-03-09T13:18:35.743+0000 7fa3c9ed3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: 0 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: 2024-03-09T13:18:36.268+0000 7fa3c9ed3640 -1 Falling back to public interface 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:40.345 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: 1 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: 2 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: 2024-03-09T13:18:39.018+0000 7fa3c9ed3640 -1 osd.1 69 log_to_monitors true 2024-03-09T13:18:40.346 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168729: 3 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168729: osd.1 up in weight 1 up_from 76 up_thru 66 down_aovery_ops 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168731: 2024-03-09T13:18:35.697+0000 7f103b791640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168731: 2024-03-09T13:18:35.699+0000 7f103b791640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:168731: 2024-03-09T13:18:35.700+0000 7f103b791640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: 0 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: 2024-03-09T13:18:36.217+0000 7f103b791640 -1 Falling back to public interface 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: 1 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: 2 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: 2024-03-09T13:18:38.792+0000 7f103b791640 -1 osd.2 69 log_to_monitors true 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:18:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:168731: 3 2024-03-09T13:18:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:18:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:168731: osd.2 up in weight 1 up_from 75 up_thru 75 down_at 70 last_clean_interval [66,69) [v2:127.0.0.1:6810/4265014776,v1:127.0.0.1:6811/4265014776] [v2:127.0.0.1:6812/4265014776,v1:127.0.0.1:6813/4265014776] exists,up 4b6cd622-d0ea-48a0-93d3-25bde8cbd51a 2024-03-09T13:18:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:18:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:18:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:18:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:18:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:18:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:18:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:18:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:18:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:168731: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:18:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:168731: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:18:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:18:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: 1 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: 2 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: 3' 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836529 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836529 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529' 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514498 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514498 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-326417514498' 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stat 70 last_clean_interval [65,69) [v2:127.0.0.1:6818/1621142875,v1:127.0.0.1:6819/1621142875] [v2:127.0.0.1:6820/1621142875,v1:127.0.0.1:6821/1621142875] exists,up 55ce8a92-5cd3-49b9-823d-afa9a56afea9 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:18:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:18:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: 1 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: 2 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: 3' 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836530 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836530 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530' 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514499 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514499 2024-03-09T13:18:41.197 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-326417514499' 2024-03-09T13:18:41.198 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:43.520 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:18:43.520 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547203 2024-03-09T13:18:43.520 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547203 2024-03-09T13:18:43.520 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-326417514498 2-322122547203' 2024-03-09T13:18:43.520 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:43.520 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070475 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070475 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836529 1-326417514498 2-322122547203 3-253403070475' 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836529 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836529 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836529 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: waiting osd.0 seq 21474836529 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:43.521 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836529 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836529 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-326417514498 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-326417514498 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514498 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 326417514498' 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: waiting osd.1 seq 326417514498 2024-03-09T13:18:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:18:43.523 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514498 2024-03-09T13:18:43.525 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in ts: ceph tell osd.2 flush_pg_stats 2024-03-09T13:18:43.525 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547202 2024-03-09T13:18:43.525 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547202 2024-03-09T13:18:43.525 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-326417514499 2-322122547202' 2024-03-09T13:18:43.525 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:43.525 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:18:43.525 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=253403070476 2024-03-09T13:18:43.525 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 253403070476 2024-03-09T13:18:43.525 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836530 1-326417514499 2-322122547202 3-253403070476' 2024-03-09T13:18:43.525 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:43.525 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836530 2024-03-09T13:18:43.525 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836530 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836530 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: waiting osd.0 seq 21474836530 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836530 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836530 -lt 21474836530 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:43.526 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-326417514499 2024-03-09T13:18:43.527 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:43.527 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:18:43.527 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-326417514499 2024-03-09T13:18:43.527 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:43.527 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514499 2024-03-09T13:18:43.527 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 326417514499' 2024-03-09T13:18:43.527 INFO:tasks.workunit.client.0.smithi162.stderr:168729: waiting osd.1 seq 326417514499 2024-03-09T13:18:43.527 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:18:43.527 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514499 2024-03-09T13:18:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547203 2024-03-09T13:18:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:18:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547203 2024-03-09T13:18:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547203 2024-03-09T13:18:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547203' 2024-03-09T13:18:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:168731: waiting osd.2 seq 322122547203 2024-03-09T13:18:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:18:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547203 2024-03-09T13:18:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070475 2024-03-09T13:18:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:44.740 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:18:44.741 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070475 2024-03-09T13:18:44.741 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:44.741 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070475 2024-03-09T13:18:44.741 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070475' 2024-03-09T13:18:44.741 INFO:tasks.workunit.client.0.smithi162.stderr:168731: waiting osd.3 seq 253403070475 2024-03-09T13:18:44.741 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:18:44.742 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070476 -lt 253403070475 2024-03-09T13:18:44.742 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:18:44.742 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:44.742 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:44.742 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:18:44.742 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:18:44.743 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:18:44.743 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:18:44.743 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:18:44.743 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:18:44.743 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:18:44.743 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:18:44.744 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:18:44.744 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:18:44.851 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephte$seqs 2024-03-09T13:18:44.851 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547202 2024-03-09T13:18:44.851 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:18:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547202 2024-03-09T13:18:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547202 2024-03-09T13:18:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547202' 2024-03-09T13:18:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:168729: waiting osd.2 seq 322122547202 2024-03-09T13:18:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:18:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547203 -lt 322122547202 2024-03-09T13:18:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-253403070476 2024-03-09T13:18:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:18:44.852 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-253403070476 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=253403070476 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 253403070476' 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: waiting osd.3 seq 253403070476 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 253403070476 -lt 253403070476 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:18:44.853 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:18:44.854 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:18:44.854 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:18:44.854 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:18:44.854 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:18:44.854 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:18:46.218 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:46.218 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:46.218 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:18:46.218 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T13:18:46.218 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:18:46.218 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T13:18:46.218 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:18:46.219 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:18:46.219 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:18:46.219 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:18:46.219 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:18:46.219 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:18:46.219 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:18:46.219 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:18:46.219 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:18:46.219 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:18:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:18:46.221 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:18:46.365 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntust/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:46.365 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:46.365 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:18:46.365 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T13:18:46.365 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:18:46.366 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T13:18:46.366 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:18:46.366 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:18:46.366 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:18:46.366 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:18:46.366 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:18:46.366 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:18:46.366 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:18:46.366 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:18:46.366 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:18:46.366 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:18:46.366 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:18:46.366 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:18:46.367 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:46.367 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:46.367 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:18:46.367 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:18:46.367 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:18:46.367 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:18:46.367 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:18:46.367 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:18:46.367 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:18:46.367 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:18:46.367 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:18:46.367 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:18:46.367 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:18:47.055 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:18:47.055 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:18:47.055 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:18:47.055 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:18:47.055 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:18:47.055 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:18:47.055 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:18:47.055 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:18:47.055 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:18:47.056 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:18:47.056 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:18:47.056 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:18:47.056 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:18:47.056 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:47.056 INFO:tasks.workunit.client.0.smithi162.stderr:168731: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:47.056 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:18:47.056 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:18:47.056 INFO:tasks.workunit.client.0.smithi162.stderr:168731: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:18:47.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:18:47.227 INFO:tasks.workunit.client.0.smithi162.stderr:/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:18:47.227 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:18:47.228 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:18:47.228 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:18:47.228 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:18:47.228 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:18:47.228 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:18:47.228 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:18:47.228 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:18:47.228 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:18:47.228 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:18:47.228 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:18:47.228 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:18:47.228 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:47.228 INFO:tasks.workunit.client.0.smithi162.stderr:168729: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:18:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:18:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:168729: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:18:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:18:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:18:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 168728 2024-03-09T13:18:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:18:47.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:18:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:18:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T13:18:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:18:47.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:18:47.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:18:47.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-03-09T13:18:47.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T13:18:47.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T13:18:47.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-09T13:18:47.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-03-09T13:18:47.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T13:18:47.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:18:47.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:18:47.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:18:47.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 171649"' 2024-03-09T13:18:47.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 171649' 2024-03-09T13:18:47.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:18:47.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:18:47.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:18:47.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:18:47.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 171651"' 2024-03-09T13:18:47.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 171651' 2024-03-09T13:18:47.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T13:18:47.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 171649 171651' 2024-03-09T13:18:47.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:18:47.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:18:47.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 171649 2024-03-09T13:18:47.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/171652: /' 2024-03-09T13:18:47.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:18:47.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/171654: /' 2024-03-09T13:18:49.821 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:18:49.821 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:18:49.821 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:18:49.821 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:18:49.821 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:18:49.821 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:18:49.821 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:18:49.821 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:18:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:18:49.823 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T13:18:49.823 INFO:tasks.workunit.client.0.smithi162.stderr:171654: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:18:49.823 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:18:49.823 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:18:49.823 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:18:49.823 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:18:49.823 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:18:49.823 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:18:49.823 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:18:49.823 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:18:49.824 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:18:49.824 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:18:49.824 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:18:49.825 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:18:49.826 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:18:49.826 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:18:49.826 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:18:49.826 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:18:49.826 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:18:49.826 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:18:49.826 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:18:49.826 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:18:49.826 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:18:49.826 INFO:tasks.workunit.client.0.smithi162.stderr:171654: start osd.1 2024-03-09T13:18:49.826 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:18:50.163 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:18:50.163 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:18:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:18:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:18:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:18:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:18:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:18:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:18:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:18:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:18:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:18:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:18:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:18:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:18:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:18:50.165 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:18:50.166 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:18:50.166 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:18:50.166 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:18:50.166 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:18:50.166 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:18:50.166 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:18:50.166 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:18:50.166 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:18:50.166 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:18:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:18:50.168 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:18:50.168 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:18:50.168 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:18:50.168 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:18:50.168 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:18:50.168 INFO:tasks.workunit.client.0.smithi162.stderr:171652: start osd.3 2024-03-09T13:18:50.168 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:18:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:18:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:18:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:18:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:18:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:18:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:171654: 2024-03-09T13:18:49.855+0000 7f78fec11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:171654: 2024-03-09T13:18:49.857+0000 7f78fec11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:171654: 2024-03-09T13:18:49.858+0000 7f78fec11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:18:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:18:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:18:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:18:54.540 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: 0 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: 2024-03-09T13:18:50.373+0000 7f78fec11640 -1 Falling back to public interface 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: 1 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:18:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: 2024-03-09T13:18:52.668+0000 7f78fec11640 -1 osd.1 78 log_to_monitors true 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: 2 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: 3 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:54.542 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:18:54.745 INFO:tasks.workunit.client.0.smithi162.stderr:171654: osd.1 up in weight 1 up_from 83 up_thru 83 down_at 79 last_171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:18:54.745 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:18:54.745 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:18:54.745 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:18:54.745 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:18:54.745 INFO:tasks.workunit.client.0.smithi162.stderr:171652: 2024-03-09T13:18:50.198+0000 7f9374611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: 2024-03-09T13:18:50.200+0000 7f9374611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: 2024-03-09T13:18:50.201+0000 7f9374611640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: 0 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: 2024-03-09T13:18:50.730+0000 7f9374611640 -1 Falling back to public interface 2024-03-09T13:18:54.746 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: 1 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: 2 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:18:54.747 INFO:tasks.workunit.client.0.smithi162.stderr:171652: 2024-03-09T13:18:53.345+0000 7f9374611640 -1 osd.3 78 log_to_monitors true 2024-03-09T13:18:54.748 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:18:54.748 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:18:54.748 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:18:54.748 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:18:54.748 INFO:tasks.workunit.client.0.smithi162.stderr:171652: 3 2024-03-09T13:18:54.748 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:18:54.748 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:18:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:171652: osd.3 up in weight 1 up_from 86 up_thru 76 down_at 79 last_clean_interval [76,78) [v2:127.0.0.1:6818/3017196539,v1:127.0.0.1:6819/3017196539] [v2:127.0.0.1:6820/3017196539,v1:127.0.0.1:6821/3017196539] exists,up 55ce8a92-5cd3-49b9-823d-afa9a56afea9 2024-03-09T13:18:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:18:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:18:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:18:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:18:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:18:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:18:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:18:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:18:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:171654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:18:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:171654: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:18:55.287 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:18:55.288 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:18:55.288 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:18:55.288 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:18:55.288 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:18:55.288 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:18:55.288 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:18:55.288 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:18:55.288 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:18:55.288 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:18:55.288 INFO:tasks.workunit.client.0.smithi162.stderr:171654: 1 2024-03-09T13:18:55.288 INFO:tasks.workunit.client.0.smithi162.stderr:171654: 2 2024-03-09T13:18:55.288 INFO:tasks.workunit.client.0.smithi162.stderr:171654: 3' 2024-03-09T13:18:55.289 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:18:55.289 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:55.289 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:18:55.289 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836534 2024-03-09T13:18:55.289 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836534 2024-03-09T13:18:55.289 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534' 2024-03-09T13:18:55.289 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:55.289 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:18:55.289 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285570 2024-03-09T13:18:55.289 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285570 2024-03-09T13:18:55.290 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-356482285570' 2024-03-09T13:18:55.290 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:55.543 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [59,78) [v2:127.0.0.1:6826/3685235003,v1:127.0.0.1:6827/3685235003] [v2:127.0.0.1:6828/3685235003,v1:127.0.0.1:6829/3685235003] exists,up d1331742-688f-482e-bfcf-9c00df7cdfa3 2024-03-09T13:18:55.543 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:18:55.543 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:18:55.543 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:18:55.543 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:18:55.543 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:18:55.543 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:18:55.543 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:18:55.543 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:18:55.543 INFO:tasks.workunit.client.0.smithi162.stderr:171652: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:18:55.543 INFO:tasks.workunit.client.0.smithi162.stderr:171652: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:18:55.544 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:18:55.544 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:18:55.544 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:18:55.544 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:18:55.544 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:18:55.544 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:18:55.544 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:18:55.544 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:18:55.544 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:18:55.544 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:18:55.544 INFO:tasks.workunit.client.0.smithi162.stderr:171652: 1 2024-03-09T13:18:55.544 INFO:tasks.workunit.client.0.smithi162.stderr:171652: 2 2024-03-09T13:18:55.545 INFO:tasks.workunit.client.0.smithi162.stderr:171652: 3' 2024-03-09T13:18:55.545 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:18:55.545 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:55.545 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:18:55.545 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836535 2024-03-09T13:18:55.545 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836535 2024-03-09T13:18:55.545 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535' 2024-03-09T13:18:55.545 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:55.545 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:18:55.545 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285571 2024-03-09T13:18:55.545 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285571 2024-03-09T13:18:55.545 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-356482285571' 2024-03-09T13:18:55.545 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:57.403 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:18:57.403 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547207 2024-03-09T13:18:57.403 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547207 2024-03-09T13:18:57.403 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-356482285570 2-322122547207' 2024-03-09T13:18:57.403 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:57.403 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:18:57.404 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=369367187458 2024-03-09T13:18:57.404 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 369367187458 2024-03-09T13:18:57.404 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836534 1-356482285570 2-322122547207 3-369367187458' 2024-03-09T13:18:57.404 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:57.404 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836534 2024-03-09T13:18:57.404 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:57.404 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:18:57.404 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836534 2024-03-09T13:18:57.404 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:57.404 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836534 2024-03-09T13:18:57.404 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-03-09T13:18:57.404 INFO:tasks.workunit.client.0.smithi162.stderr:171654: waiting osd.0 seq 21474836534 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836534 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836534 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-356482285570 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-356482285570 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285570 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 356482285570' 2024-03-09T13:18:57.405 INFO:tasks.workunit.client.0.smithi162.stderr:171654: waiting osd.1 seq 356482285570 2024-03-09T13:18:57.406 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:18:57.406 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285571 -lt 356482285570 2024-03-09T13:18:57.406 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:57.735 INFO:tasks.workunit.client.0.smithi162.stderr:1716tell osd.2 flush_pg_stats 2024-03-09T13:18:57.735 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547208 2024-03-09T13:18:57.735 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547208 2024-03-09T13:18:57.735 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-356482285571 2-322122547208' 2024-03-09T13:18:57.735 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:18:57.735 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:18:57.735 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=369367187459 2024-03-09T13:18:57.735 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 369367187459 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836535 1-356482285571 2-322122547208 3-369367187459' 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836535 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836535 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836535 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: waiting osd.0 seq 21474836535 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836533 -lt 21474836535 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:18:57.736 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:18:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836535 -lt 21474836535 2024-03-09T13:18:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-356482285571 2024-03-09T13:18:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:18:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-356482285571 2024-03-09T13:18:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285571 2024-03-09T13:18:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 356482285571' 2024-03-09T13:18:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:171652: waiting osd.1 seq 356482285571 2024-03-09T13:18:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:18:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285571 -lt 356482285571 2024-03-09T13:18:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:58.718 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547207 2024-03-09T13:18:58.718 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:58.718 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:18:58.718 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547207 2024-03-09T13:18:58.718 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:58.718 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547207 2024-03-09T13:18:58.718 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547207' 2024-03-09T13:18:58.718 INFO:tasks.workunit.client.0.smithi162.stderr:171654: waiting osd.2 seq 322122547207 2024-03-09T13:18:58.718 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:18:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547208 -lt 322122547207 2024-03-09T13:18:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-369367187458 2024-03-09T13:18:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:18:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-369367187458 2024-03-09T13:18:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=369367187458 2024-03-09T13:18:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 369367187458' 2024-03-09T13:18:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:171654: waiting osd.3 seq 369367187458 2024-03-09T13:18:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:18:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 369367187459 -lt 369367187458 2024-03-09T13:18:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:18:58.720 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:58.720 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:58.720 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:18:58.720 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:18:58.720 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:18:58.720 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:18:58.720 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:18:58.720 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:18:58.720 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:18:58.720 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:18:58.720 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:18:58.720 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:18:59.023 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.c52: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547208 2024-03-09T13:18:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:18:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547208 2024-03-09T13:18:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547208 2024-03-09T13:18:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547208' 2024-03-09T13:18:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:171652: waiting osd.2 seq 322122547208 2024-03-09T13:18:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:18:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547208 -lt 322122547208 2024-03-09T13:18:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:18:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-369367187459 2024-03-09T13:18:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-369367187459 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=369367187459 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 369367187459' 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: waiting osd.3 seq 369367187459 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 369367187459 -lt 369367187459 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:18:59.025 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:18:59.026 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:18:59.026 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:18:59.026 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:18:59.026 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:18:59.026 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:18:59.026 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:19:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:19:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T13:19:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:19:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T13:19:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:19:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:19:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:19:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:19:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:19:00.232 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:19:00.233 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:19:00.234 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:19:00.234 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:19:00.234 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:19:00.234 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-09T13:19:00.234 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-09T13:19:00.497 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.lient.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:00.497 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:00.497 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:19:00.497 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T13:19:00.497 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:19:00.498 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:00.499 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:00.499 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:19:00.499 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:19:00.499 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:19:00.499 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:19:00.499 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:19:00.499 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:19:00.499 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:19:00.499 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:19:00.499 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:19:00.499 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-09T13:19:00.499 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-09T13:19:01.037 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T13:19:01.037 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:19:01.037 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:19:01.037 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:19:01.037 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:19:01.037 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:19:01.037 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:19:01.037 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:19:01.037 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:19:01.037 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:19:01.038 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:19:01.038 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:19:01.038 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:19:01.038 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:01.038 INFO:tasks.workunit.client.0.smithi162.stderr:171654: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:01.038 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:19:01.038 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:19:01.038 INFO:tasks.workunit.client.0.smithi162.stderr:171654: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:19:01.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:19:01.262 INFO:tasks.workunit.client.0.smithi162.stderr:client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T13:19:01.262 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:19:01.262 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:19:01.263 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr:171652: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 171651 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:19:01.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:19:01.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T13:19:01.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T13:19:01.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:19:01.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:19:01.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:19:01.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:19:01.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:19:01.593 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:19:01.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:18:31.073525+0000 2024-03-09T13:19:01.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:19:02.120 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:19:02.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:18:31.073525+0000 2024-03-09T13:19:02.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:19:02.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:18:31.073525+0000 2024-03-09T13:19:02.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:19:02.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:19:02.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:19:02.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:19:02.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:19:02.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:19:02.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:19:02.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:19:02.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:18:31.073525+0000 '>' 2024-03-09T13:18:31.073525+0000 2024-03-09T13:19:02.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:19:03.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:19:03.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:19:03.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:19:03.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:19:03.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:19:03.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:19:03.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:19:03.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:18:31.073525+0000 '>' 2024-03-09T13:18:31.073525+0000 2024-03-09T13:19:03.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:19:04.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:19:04.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:19:04.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:19:04.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:19:04.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:19:04.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:19:04.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:19:04.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:19:02.462671+0000 '>' 2024-03-09T13:18:31.073525+0000 2024-03-09T13:19:04.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:19:04.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T13:19:04.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:19:04.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:19:04.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:19:04.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 174695"' 2024-03-09T13:19:04.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 174695' 2024-03-09T13:19:04.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:19:04.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:19:04.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:19:04.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 174697"' 2024-03-09T13:19:04.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 174697' 2024-03-09T13:19:04.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:19:04.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T13:19:04.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 174695 174697' 2024-03-09T13:19:04.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:19:04.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:19:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 174695 2024-03-09T13:19:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/174698: /' 2024-03-09T13:19:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:19:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/174700: /' 2024-03-09T13:19:07.465 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:19:07.465 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:19:07.465 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:19:07.465 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:19:07.465 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:19:07.465 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:19:07.465 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:19:07.465 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:19:07.465 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:19:07.465 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:19:07.465 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T13:19:07.466 INFO:tasks.workunit.client.0.smithi162.stderr:174698: _ 2024-03-09T13:19:07.467 INFO:tasks.workunit.client.0.smithi162.stderr:174698: hinfo_key 2024-03-09T13:19:07.467 INFO:tasks.workunit.client.0.smithi162.stderr:174698: snapset 2024-03-09T13:19:07.467 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:19:07.467 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:19:07.467 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:19:07.467 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:19:07.467 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:19:07.467 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:19:07.467 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:19:07.468 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:19:07.468 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:19:07.468 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:19:07.469 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:19:07.469 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:19:07.469 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:19:07.470 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:19:07.470 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:19:07.470 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:19:07.470 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:19:07.471 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:19:07.471 INFO:tasks.workunit.client.0.smithi162.stderr:174698: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:19:07.471 INFO:tasks.workunit.client.0.smithi162.stderr:174698: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:19:07.471 INFO:tasks.workunit.client.0.smithi162.stderr:174698: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:19:07.471 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:19:07.471 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:19:07.471 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:19:07.471 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:19:07.472 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:19:07.472 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:19:07.472 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:19:07.472 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:19:07.472 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:19:07.472 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:19:07.472 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:19:07.472 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:19:07.472 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:19:07.472 INFO:tasks.workunit.client.0.smithi162.stderr:174698: start osd.3 2024-03-09T13:19:07.819 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:19:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:19:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:19:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:19:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:19:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:19:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:19:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:19:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:19:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:19:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:19:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:19:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:19:07.820 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:19:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:19:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:19:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:19:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:19:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:19:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:19:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:19:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:19:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T13:19:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:174700: _ 2024-03-09T13:19:07.821 INFO:tasks.workunit.client.0.smithi162.stderr:174700: hinfo_key 2024-03-09T13:19:07.822 INFO:tasks.workunit.client.0.smithi162.stderr:174700: snapset 2024-03-09T13:19:07.822 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:19:07.822 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:19:07.822 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:19:07.822 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:19:07.822 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:19:07.822 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:19:07.822 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:19:07.822 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:19:07.822 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:19:07.823 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:19:07.824 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:19:07.824 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:19:07.824 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:19:07.824 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:19:07.824 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:19:07.824 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:19:07.824 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:19:07.824 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:19:07.824 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:19:07.824 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:19:07.824 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:19:07.824 INFO:tasks.workunit.client.0.smithi162.stderr:174700: start osd.1 2024-03-09T13:19:11.647 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:19:11.647 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:19:11.647 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:19:11.647 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:19:11.647 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:19:11.647 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:19:11.647 INFO:tasks.workunit.client.0.smithi162.stderr:174698: 2024-03-09T13:19:07.498+0000 7ff337ea8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:11.647 INFO:tasks.workunit.client.0.smithi162.stderr:174698: 2024-03-09T13:19:07.500+0000 7ff337ea8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:11.647 INFO:tasks.workunit.client.0.smithi162.stderr:174698: 2024-03-09T13:19:07.502+0000 7ff337ea8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:11.647 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: 0 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: 2024-03-09T13:19:08.020+0000 7ff337ea8640 -1 Falling back to public interface 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:11.648 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: 1 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: 2 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: 2024-03-09T13:19:10.369+0000 7ff337ea8640 -1 osd.3 87 log_to_monitors true 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:11.649 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:11.650 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:19:11.650 INFO:tasks.workunit.client.0.smithi162.stderr:174698: 3 2024-03-09T13:19:11.650 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:12.598 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-03-09T13:19:12.598 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:19:12.598 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:19:12.598 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:19:12.598 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: 2024-03-09T13:19:07.850+0000 7fe751381640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: 2024-03-09T13:19:07.879+0000 7fe751381640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: 2024-03-09T13:19:07.880+0000 7fe751381640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: 0 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:12.599 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:19:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:174700: 2024-03-09T13:19:08.402+0000 7fe751381640 -1 Falling back to public interface 2024-03-09T13:19:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:19:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:174700: 1 2024-03-09T13:19:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:19:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:19:12.600 INFO:tasks.workunit.client.0.smithi162.stderr:174700: 2 2024-03-09T13:19:12.601 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:12.601 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:19:12.601 INFO:tasks.workunit.client.0.smithi162.stderr:174700: 2024-03-09T13:19:11.066+0000 7fe751381640 -1 osd.1 87 log_to_monitors true 2024-03-09T13:19:12.601 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:12.601 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:12.601 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:12.601 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:19:12.601 INFO:tasks.workunit.client.0.smithi162.stderr:174700: 3 2024-03-09T13:19:12.601 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:12.601 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:19:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:174700: osd.1 up in weight 1 up_from 94 up_thru 83 down_as.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:19:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:174698: osd.3 up in weight 1 up_from 92 up_thru 92 down_at 88 last_clean_interval [86,87) [v2:127.0.0.1:6818/1279434216,v1:127.0.0.1:6819/1279434216] [v2:127.0.0.1:6820/1279434216,v1:127.0.0.1:6821/1279434216] exists,up d1331742-688f-482e-bfcf-9c00df7cdfa3 2024-03-09T13:19:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:19:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:19:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:19:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:19:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:19:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:19:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:19:12.799 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: 1 2024-03-09T13:19:12.800 INFO:tasks.workunit.client.0.smithi162.stderr:174698: 2 2024-03-09T13:19:12.801 INFO:tasks.workunit.client.0.smithi162.stderr:174698: 3' 2024-03-09T13:19:12.801 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:19:12.801 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:19:12.801 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:19:12.801 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836539 2024-03-09T13:19:12.801 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836539 2024-03-09T13:19:12.801 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539' 2024-03-09T13:19:12.801 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:19:12.801 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:19:12.801 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925826 2024-03-09T13:19:12.801 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925826 2024-03-09T13:19:12.801 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-403726925826' 2024-03-09T13:19:13.465 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd it 88 last_clean_interval [83,87) [v2:127.0.0.1:6826/2578281659,v1:127.0.0.1:6827/2578281659] [v2:127.0.0.1:6828/2578281659,v1:127.0.0.1:6829/2578281659] exists,up 55ce8a92-5cd3-49b9-823d-afa9a56afea9 2024-03-09T13:19:13.465 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:19:13.465 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:19:13.465 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:19:13.465 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:19:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:19:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:19:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:19:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:19:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:174700: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:19:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:174700: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:19:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:19:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:19:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:19:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:19:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:19:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:19:13.466 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: 1 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: 2 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: 3' 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836541 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836541 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541' 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=403726925827 2024-03-09T13:19:13.467 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 403726925827 2024-03-09T13:19:13.468 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-403726925827' 2024-03-09T13:19:13.468 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:19:15.332 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stan $ids 2024-03-09T13:19:15.332 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:19:15.332 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547212 2024-03-09T13:19:15.332 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547212 2024-03-09T13:19:15.332 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-403726925826 2-322122547212' 2024-03-09T13:19:15.332 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:19:15.332 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:19:15.332 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991234 2024-03-09T13:19:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991234 2024-03-09T13:19:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-403726925826 2-322122547212 3-395136991234' 2024-03-09T13:19:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:19:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836539 2024-03-09T13:19:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:19:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:19:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836539 2024-03-09T13:19:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:19:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836539 2024-03-09T13:19:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-03-09T13:19:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:174698: waiting osd.0 seq 21474836539 2024-03-09T13:19:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:19:15.333 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836539 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836539 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-403726925826 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-403726925826 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925826 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 403726925826' 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: waiting osd.1 seq 403726925826 2024-03-09T13:19:15.334 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:19:15.335 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925826 2024-03-09T13:19:15.543 INFO:tasks.workunit.client.0.smithi162.stderr:174698ts: ceph tell osd.2 flush_pg_stats 2024-03-09T13:19:15.543 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=322122547213 2024-03-09T13:19:15.543 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 322122547213 2024-03-09T13:19:15.543 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-403726925827 2-322122547213' 2024-03-09T13:19:15.543 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:19:15.543 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:19:15.543 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991235 2024-03-09T13:19:15.543 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991235 2024-03-09T13:19:15.543 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836541 1-403726925827 2-322122547213 3-395136991235' 2024-03-09T13:19:15.543 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:19:15.543 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836541 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836541 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836541 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: waiting osd.0 seq 21474836541 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836538 -lt 21474836541 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836541 -lt 21474836541 2024-03-09T13:19:15.544 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:19:15.545 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-403726925827 2024-03-09T13:19:15.545 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:19:15.545 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:19:15.545 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-403726925827 2024-03-09T13:19:15.545 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:19:15.545 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=403726925827 2024-03-09T13:19:15.545 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 403726925827' 2024-03-09T13:19:15.545 INFO:tasks.workunit.client.0.smithi162.stderr:174700: waiting osd.1 seq 403726925827 2024-03-09T13:19:15.545 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:19:15.545 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 403726925827 -lt 403726925827 2024-03-09T13:19:16.586 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in : /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:19:16.586 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547212 2024-03-09T13:19:16.586 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:19:16.586 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:19:16.586 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547212 2024-03-09T13:19:16.586 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:19:16.586 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547212 2024-03-09T13:19:16.586 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547212' 2024-03-09T13:19:16.586 INFO:tasks.workunit.client.0.smithi162.stderr:174698: waiting osd.2 seq 322122547212 2024-03-09T13:19:16.586 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547214 -lt 322122547212 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991234 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991234 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991234 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991234' 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: waiting osd.3 seq 395136991234 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991235 -lt 395136991234 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:19:16.587 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:19:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:19:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:19:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:19:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:19:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:19:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:19:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:19:16.588 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:19:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.c$seqs 2024-03-09T13:19:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-322122547213 2024-03-09T13:19:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:19:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:19:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-322122547213 2024-03-09T13:19:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:19:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=322122547213 2024-03-09T13:19:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 322122547213' 2024-03-09T13:19:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:174700: waiting osd.2 seq 322122547213 2024-03-09T13:19:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:19:16.797 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 322122547214 -lt 322122547213 2024-03-09T13:19:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:19:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-395136991235 2024-03-09T13:19:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:19:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:19:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-395136991235 2024-03-09T13:19:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:19:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991235 2024-03-09T13:19:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 395136991235' 2024-03-09T13:19:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:174700: waiting osd.3 seq 395136991235 2024-03-09T13:19:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:19:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991235 -lt 395136991235 2024-03-09T13:19:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:19:16.798 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:16.799 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:16.799 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:19:16.799 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:19:16.799 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:19:16.799 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:19:16.799 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:19:16.799 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:19:16.799 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:19:16.799 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:19:16.799 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:19:16.799 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:19:18.082 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtelient.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:19:18.082 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:18.082 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:18.082 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T13:19:18.082 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T13:19:18.082 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:19:18.082 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T13:19:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:19:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:19:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:19:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:19:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:19:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:19:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:19:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:19:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:19:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:19:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:19:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:19:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:18.084 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:18.084 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T13:19:18.084 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T13:19:18.084 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:19:18.084 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:19:18.084 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:19:18.084 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:19:18.084 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:19:18.084 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:19:18.084 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:19:18.084 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-09T13:19:18.288 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalonst/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:18.288 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:18.288 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T13:19:18.288 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T13:19:18.288 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:19:18.288 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T13:19:18.288 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:19:18.288 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:19:18.288 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:19:18.288 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:19:18.288 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:19:18.288 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:19:18.289 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:19:18.289 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:19:18.289 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:19:18.289 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:19:18.289 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:19:18.289 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:19:18.289 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:18.289 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:18.289 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T13:19:18.289 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T13:19:18.289 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:19:18.289 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:19:18.289 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:19:18.290 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:19:18.290 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:19:18.290 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:19:18.290 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:19:18.290 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-09T13:19:18.290 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-09T13:19:19.023 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T13:19:19.023 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:19:19.023 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:19:19.023 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:19:19.023 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:19:19.023 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:19:19.023 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:19:19.023 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:19:19.024 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:19:19.024 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:19:19.024 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:19:19.024 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:19:19.024 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:19:19.024 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:19.024 INFO:tasks.workunit.client.0.smithi162.stderr:174700: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:19.024 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:19:19.024 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:19:19.024 INFO:tasks.workunit.client.0.smithi162.stderr:174700: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:19:19.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:19:19.282 INFO:tasks.workunit.client.0.smithi162.stderr:e/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-09T13:19:19.282 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T13:19:19.282 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:19:19.282 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:19:19.282 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:19:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:19:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:19:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:19:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:19:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:19:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:19:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:19:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:19:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:19:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T13:19:19.283 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T13:19:19.284 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:19:19.284 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:19:19.284 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:19:19.284 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:19:19.284 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:19:19.284 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:19:19.284 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:19:19.284 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-09T13:19:19.284 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-09T13:19:19.284 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T13:19:19.284 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:19:19.284 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:19:19.284 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:19:19.926 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:19:19.926 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:19:19.926 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:174698: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:174698: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 174697 2024-03-09T13:19:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:19:19.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:19:19.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:19:19.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T13:19:19.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:19:19.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:19:19.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:19:19.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:19:19.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:19:19.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:19:19.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:19:19.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:19:19.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:19:19.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:19:19.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:19:19.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:19:20.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:19:20.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:19:20.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:19:20.080 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:19:20.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:19:20.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:19:20.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:19:20.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:19:20.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:19:20.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:19:20.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:19:20.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:19:20.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:19:20.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:19:20.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:19:20.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:19:20.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:19:20.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:19:20.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:19:20.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:19:20.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:19:20.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:19:20.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:19:20.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:19:20.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:19:20.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:19:20.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:19:20.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:19:20.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:19:20.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:19:20.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:19:20.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:19:20.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:19:20.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:19:20.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:19:20.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:19:20.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:19:20.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:19:20.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:19:20.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:19:20.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:19:20.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:19:20.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:19:20.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:19:20.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:19:20.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:19:20.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:19:20.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:19:20.124 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:19:20.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:19:20.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:19:20.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:19:20.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:19:20.126 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:19:20.126 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:19:20.126 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:19:20.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:19:20.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:19:20.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:19:20.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:19:20.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:19:20.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2024-03-09T13:19:20.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2024-03-09T13:19:20.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:820: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2024-03-09T13:19:20.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:798: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2024-03-09T13:19:20.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:799: corrupt_and_repair_jerasure: local allow_overwrites=true 2024-03-09T13:19:20.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:800: corrupt_and_repair_jerasure: local poolname=ecpool 2024-03-09T13:19:20.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:802: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2024-03-09T13:19:20.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:19:20.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:19:20.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:19:20.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:19:20.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:19:20.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T13:19:20.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:19:20.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:19:20.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:19:20.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:19:20.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:19:20.600 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:19:20.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:19:20.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T13:19:20.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:19:20.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:19:20.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:19:20.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:19:20.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:19:20.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:19:20.802 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:19:20.802 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:19:20.802 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:19:20.803 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:19:20.803 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:19:20.803 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:19:20.803 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:19:20.803 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:19:20.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:19:20.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:19:20.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:19:20.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:19:20.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:19:20.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:19:20.884 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:19:20.884 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:19:20.884 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:19:20.884 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:19:20.884 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:19:20.884 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:19:20.884 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:19:20.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:19:20.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:19:20.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:803: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2024-03-09T13:19:20.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:19:20.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:19:20.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:19:20.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:19:20.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:19:20.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:19:21.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:19:21.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:19:21.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:19:21.762 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:19:21.762 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:19:21.762 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:19:21.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:19:21.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:19:21.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:19:21.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: seq 0 3 2024-03-09T13:19:21.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T13:19:21.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2024-03-09T13:19:21.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:19:21.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:19:21.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:19:21.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:19:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:19:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:19:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:19:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:19:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:19:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:19:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:19:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:19:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:19:21.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:19:21.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:19:21.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:19:21.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:19:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:19:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:19:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:19:21.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:19:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:19:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:19:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:19:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:19:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:19:21.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:19:21.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:19:21.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:19:21.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:19:21.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:19:21.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:19:21.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:19:21.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:19:21.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:19:21.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e607867-4be0-467b-8b7d-034b8a0f162d 2024-03-09T13:19:21.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9e607867-4be0-467b-8b7d-034b8a0f162d' 2024-03-09T13:19:21.798 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 9e607867-4be0-467b-8b7d-034b8a0f162d 2024-03-09T13:19:21.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:19:21.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDZYexlINVyMBAAe7B7+J2qt3q/JRzqWIsHPw== 2024-03-09T13:19:21.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDZYexlINVyMBAAe7B7+J2qt3q/JRzqWIsHPw=="}' 2024-03-09T13:19:21.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e607867-4be0-467b-8b7d-034b8a0f162d -i td/osd-scrub-repair/0/new.json 2024-03-09T13:19:22.016 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:19:22.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:19:22.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZYexlINVyMBAAe7B7+J2qt3q/JRzqWIsHPw== --osd-uuid 9e607867-4be0-467b-8b7d-034b8a0f162d 2024-03-09T13:19:22.055 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:22.057+0000 7f9391e10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:22.067 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:22.069+0000 7f9391e10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:22.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:22.070+0000 7f9391e10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:22.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:22.071+0000 7f9391e10640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:19:22.069 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:22.071+0000 7f9391e10640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:19:22.069 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:22.071+0000 7f9391e10640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:19:22.151 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:22.153+0000 7f9391e10640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:19:27.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:19:27.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:19:27.614 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:19:27.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:19:27.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:19:27.975 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:19:27.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:19:27.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:19:27.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:19:27.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:19:27.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:19:28.000 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:28.003+0000 7f2314349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:28.013 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:28.015+0000 7f2314349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:28.014 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:28.016+0000 7f2314349640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:28.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:19:28.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:19:28.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:19:28.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:19:28.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:19:28.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:28.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:19:28.251 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:19:28.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:28.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:28.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:28.543 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:28.546+0000 7f2314349640 -1 Falling back to public interface 2024-03-09T13:19:29.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:29.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:29.524 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:19:29.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:19:29.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:29.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:29.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:30.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:30.803 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:19:30.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:30.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:19:30.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:30.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:30.869 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:30.871+0000 7f2314349640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:19:31.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:32.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:32.089 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:19:32.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:32.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:19:32.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:32.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:32.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:33.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:33.400 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:19:33.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:33.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:19:33.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:33.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:33.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:34.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:34.778 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:19:34.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:19:34.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:34.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:35.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:36.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:36.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:36.065 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:19:36.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:19:36.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:36.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:36.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:37.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:37.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:37.348 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:19:37.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:19:37.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:37.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:37.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:38.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:38.632 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:19:38.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:38.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:19:38.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:38.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:38.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:39.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:39.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:39.917 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:19:39.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:19:39.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:39.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:40.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:41.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:41.199 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:19:41.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:41.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:19:41.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:41.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:41.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:42.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:42.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:42.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:19:42.578 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:19:42.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:42.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:43.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:44.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:44.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:44.031 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:19:44.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:19:44.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:44.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:44.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:45.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:45.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:45.592 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:19:45.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:19:45.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:45.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:46.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:47.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:47.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:47.601 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:19:47.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:19:47.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:47.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:48.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:49.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:49.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:49.026 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:19:49.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:19:49.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:49.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:49.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:50.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:50.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:50.505 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:19:50.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:19:50.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:50.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:50.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:51.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:51.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:51.914 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:19:51.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:19:51.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:51.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:52.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:19:53.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:19:53.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:19:53.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:19:53.275 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:19:53.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:19:53.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:19:53.554 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2597533019,v1:127.0.0.1:6803/2597533019] [v2:127.0.0.1:6804/2597533019,v1:127.0.0.1:6805/2597533019] exists,up 9e607867-4be0-467b-8b7d-034b8a0f162d 2024-03-09T13:19:53.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:19:53.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:19:53.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:19:53.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T13:19:53.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2024-03-09T13:19:53.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:19:53.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:19:53.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:19:53.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:19:53.559 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:19:53.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:19:53.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:19:53.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:19:53.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:19:53.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9dbfada3-cce6-4508-b697-011f9f8cf024 2024-03-09T13:19:53.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9dbfada3-cce6-4508-b697-011f9f8cf024' 2024-03-09T13:19:53.561 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 9dbfada3-cce6-4508-b697-011f9f8cf024 2024-03-09T13:19:53.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:19:53.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5Yexl2SZiIhAAGVKLCWl5iC0sJM6Ub6mq1w== 2024-03-09T13:19:53.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5Yexl2SZiIhAAGVKLCWl5iC0sJM6Ub6mq1w=="}' 2024-03-09T13:19:53.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9dbfada3-cce6-4508-b697-011f9f8cf024 -i td/osd-scrub-repair/1/new.json 2024-03-09T13:19:53.933 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:19:53.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:19:53.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5Yexl2SZiIhAAGVKLCWl5iC0sJM6Ub6mq1w== --osd-uuid 9dbfada3-cce6-4508-b697-011f9f8cf024 2024-03-09T13:19:53.973 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:53.975+0000 7f984cec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:54.027 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:54.029+0000 7f984cec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:54.028 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:54.029+0000 7f984cec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:19:54.029 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:54.030+0000 7f984cec6640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:19:54.029 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:54.030+0000 7f984cec6640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:19:54.029 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:54.030+0000 7f984cec6640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:19:54.126 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:19:54.127+0000 7f984cec6640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:19:59.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:19:59.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:19:59.773 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:19:59.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:19:59.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:20:00.134 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:20:00.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:20:00.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:20:00.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:20:00.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:20:00.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:20:00.160 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:00.162+0000 7ff9a981c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:20:00.162 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:00.164+0000 7ff9a981c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:20:00.163 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:00.165+0000 7ff9a981c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:20:00.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:20:00.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:20:00.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:20:00.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:20:00.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:20:00.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:00.408 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:20:00.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:20:00.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:00.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:00.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:00.697 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:00.700+0000 7ff9a981c640 -1 Falling back to public interface 2024-03-09T13:20:01.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:01.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:01.681 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:20:01.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:20:01.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:01.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:01.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:02.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:02.998 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:20:02.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:02.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:20:02.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:02.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:03.024 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:03.026+0000 7ff9a981c640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:20:03.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:04.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:04.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:04.359 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:20:04.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:20:04.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:04.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:04.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:05.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:05.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:05.650 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:20:05.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:20:05.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:05.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:06.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:07.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:07.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:07.111 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:20:07.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:20:07.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:07.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:07.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:08.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:08.396 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:20:08.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:08.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:20:08.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:08.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:08.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:09.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:09.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:09.682 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:20:09.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:20:09.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:09.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:10.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:11.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:11.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:11.073 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:20:11.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:20:11.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:11.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:11.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:12.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:12.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:12.361 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:20:12.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:20:12.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:12.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:12.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:13.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:13.664 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:20:13.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:13.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:20:13.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:13.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:14.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:15.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:15.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:15.087 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:20:15.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:20:15.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:15.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:15.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:16.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:16.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:16.419 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:20:16.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:20:16.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:16.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:17.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:18.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:18.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:18.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:20:18.385 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:20:18.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:18.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:19.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:20.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:20.012 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:20:20.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:20.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:20:20.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:20.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:20.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:21.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:21.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:21.848 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:20:21.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:20:21.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:21.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:22.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:23.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:23.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:23.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:20:23.676 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:20:23.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:23.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:24.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:25.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:25.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:25.124 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:20:25.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:20:25.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:25.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:25.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:26.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:26.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:26.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:20:26.410 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:20:26.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:26.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:20:26.682 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1502206887,v1:127.0.0.1:6811/1502206887] [v2:127.0.0.1:6812/1502206887,v1:127.0.0.1:6813/1502206887] exists,up 9dbfada3-cce6-4508-b697-011f9f8cf024 2024-03-09T13:20:26.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:20:26.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:20:26.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:20:26.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T13:20:26.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2024-03-09T13:20:26.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:20:26.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:20:26.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T13:20:26.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:20:26.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:20:26.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:20:26.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:20:26.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:20:26.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:20:26.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:20:26.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:20:26.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:20:26.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:20:26.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:20:26.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:20:26.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:20:26.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:20:26.692 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:20:26.692 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:20:26.692 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:20:26.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:20:26.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:20:26.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:20:26.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:20:26.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:20:26.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:20:26.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:20:26.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:20:26.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:20:26.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:20:26.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:20:26.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:20:26.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:20:26.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:20:26.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:20:26.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c7f558e-01b7-4c13-abba-dec214f5e7f0 2024-03-09T13:20:26.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5c7f558e-01b7-4c13-abba-dec214f5e7f0' 2024-03-09T13:20:26.695 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 5c7f558e-01b7-4c13-abba-dec214f5e7f0 2024-03-09T13:20:26.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:20:26.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAaYuxlFmsSKhAAn5leuql93N9GoiO1S+yQkA== 2024-03-09T13:20:26.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAaYuxlFmsSKhAAn5leuql93N9GoiO1S+yQkA=="}' 2024-03-09T13:20:26.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c7f558e-01b7-4c13-abba-dec214f5e7f0 -i td/osd-scrub-repair/2/new.json 2024-03-09T13:20:27.081 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:20:27.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T13:20:27.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAaYuxlFmsSKhAAn5leuql93N9GoiO1S+yQkA== --osd-uuid 5c7f558e-01b7-4c13-abba-dec214f5e7f0 2024-03-09T13:20:27.121 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:27.122+0000 7efd2af19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:20:27.166 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:27.167+0000 7efd2af19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:20:27.167 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:27.168+0000 7efd2af19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:20:27.167 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:27.168+0000 7efd2af19640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:20:27.168 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:27.168+0000 7efd2af19640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:20:27.168 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:27.168+0000 7efd2af19640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:20:27.241 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:27.242+0000 7efd2af19640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T13:20:33.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T13:20:33.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:20:33.287 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T13:20:33.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T13:20:33.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:20:33.639 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T13:20:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T13:20:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:20:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:20:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:20:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:20:33.665 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:33.666+0000 7f0a26e89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:20:33.667 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:33.668+0000 7f0a26e89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:20:33.668 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:33.669+0000 7f0a26e89640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:20:33.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T13:20:33.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:20:33.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:20:33.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:20:33.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:20:33.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:33.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:20:33.916 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:20:33.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:33.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:34.188 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:34.189+0000 7f0a26e89640 -1 Falling back to public interface 2024-03-09T13:20:34.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:35.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:35.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:35.194 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:20:35.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:20:35.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:35.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:35.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:36.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:36.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:36.476 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:20:36.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:20:36.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:36.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:36.512 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:20:36.513+0000 7f0a26e89640 -1 osd.2 0 log_to_monitors true 2024-03-09T13:20:36.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:37.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:37.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:37.754 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:20:37.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:20:37.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:37.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:38.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:39.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:39.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:39.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:20:39.078 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:20:39.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:39.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:39.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:40.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:40.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:40.363 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:20:40.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:20:40.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:40.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:41.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:41.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:41.753 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:20:41.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:20:41.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:41.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:42.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:43.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:43.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:43.086 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:20:43.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:20:43.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:43.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:43.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:44.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:44.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:44.482 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:20:44.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:20:44.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:44.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:45.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:45.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:45.767 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:20:45.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:20:45.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:45.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:46.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:47.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:47.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:47.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:20:47.098 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:20:47.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:47.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:47.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:48.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:48.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:48.471 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:20:48.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:20:48.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:48.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:48.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:49.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:49.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:49.858 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:20:49.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:20:49.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:49.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:50.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:51.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:51.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:51.509 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:20:51.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:20:51.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:51.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:52.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:53.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:53.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:53.394 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:20:53.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:20:53.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:53.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:53.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:54.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:54.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:54.697 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:20:54.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:20:54.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:54.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:55.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:56.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:56.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:56.307 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:20:56.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:20:56.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:56.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:57.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:58.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:58.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:58.101 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:20:58.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:20:58.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:58.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:58.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:20:59.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:20:59.622 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:20:59.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:20:59.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:20:59.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:20:59.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:20:59.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:00.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:00.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:00.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:21:00.909 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:21:00.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:00.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:21:01.187 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1068551888,v1:127.0.0.1:6819/1068551888] [v2:127.0.0.1:6820/1068551888,v1:127.0.0.1:6821/1068551888] exists,up 5c7f558e-01b7-4c13-abba-dec214f5e7f0 2024-03-09T13:21:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:21:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:21:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:21:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2024-03-09T13:21:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2024-03-09T13:21:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:21:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:21:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-09T13:21:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:21:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:21:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:21:01.194 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:21:01.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:21:01.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:21:01.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:21:01.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:21:01.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:21:01.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:21:01.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:21:01.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:21:01.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:21:01.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:21:01.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:21:01.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:21:01.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:21:01.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:21:01.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:21:01.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0fe1e115-9ef0-4727-a510-86a123c7da5e 2024-03-09T13:21:01.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 0fe1e115-9ef0-4727-a510-86a123c7da5e' 2024-03-09T13:21:01.195 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 0fe1e115-9ef0-4727-a510-86a123c7da5e 2024-03-09T13:21:01.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:21:01.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9YuxlOTaDDBAA/w8O53TVee5BFxA9EBY3Hg== 2024-03-09T13:21:01.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9YuxlOTaDDBAA/w8O53TVee5BFxA9EBY3Hg=="}' 2024-03-09T13:21:01.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0fe1e115-9ef0-4727-a510-86a123c7da5e -i td/osd-scrub-repair/3/new.json 2024-03-09T13:21:01.532 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:21:01.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-09T13:21:01.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9YuxlOTaDDBAA/w8O53TVee5BFxA9EBY3Hg== --osd-uuid 0fe1e115-9ef0-4727-a510-86a123c7da5e 2024-03-09T13:21:01.572 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:01.573+0000 7f17bc547640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:21:01.618 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:01.618+0000 7f17bc547640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:21:01.618 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:01.619+0000 7f17bc547640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:21:01.618 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:01.619+0000 7f17bc547640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T13:21:01.619 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:01.619+0000 7f17bc547640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T13:21:01.619 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:01.619+0000 7f17bc547640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T13:21:01.692 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:01.693+0000 7f17bc547640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-09T13:21:07.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-09T13:21:07.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:21:07.642 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-03-09T13:21:07.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-09T13:21:07.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:21:08.001 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-03-09T13:21:08.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-09T13:21:08.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:21:08.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:21:08.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:21:08.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:21:08.027 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:08.028+0000 7fee0d2b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:21:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:08.030+0000 7fee0d2b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:21:08.030 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:08.031+0000 7fee0d2b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:21:08.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-09T13:21:08.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:21:08.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:21:08.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:21:08.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:21:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:21:08.278 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:21:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:08.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:08.556 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:08.557+0000 7fee0d2b5640 -1 Falling back to public interface 2024-03-09T13:21:09.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:09.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:09.552 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:21:09.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:21:09.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:09.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:09.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:10.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:10.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:10.833 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:21:10.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:21:10.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:10.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:10.881 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:10.882+0000 7fee0d2b5640 -1 osd.3 0 log_to_monitors true 2024-03-09T13:21:11.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:12.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:12.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:12.113 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:21:12.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:21:12.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:12.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:12.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:13.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:13.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:13.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:21:13.407 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:21:13.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:13.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:13.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:14.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:14.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:14.715 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:21:14.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:21:14.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:14.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:14.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:15.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:15.999 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:21:16.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:16.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:21:16.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:16.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:16.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:17.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:17.287 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:21:17.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:17.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:21:17.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:17.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:17.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:18.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:18.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:18.698 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:21:18.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:21:18.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:18.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:18.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:19.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:19.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:19.990 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:21:19.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:21:19.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:19.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:20.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:21.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:21.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:21.275 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:21:21.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:21:21.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:21.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:21.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:22.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:22.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:22.560 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:21:22.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:21:22.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:22.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:22.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:23.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:23.844 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:21:23.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:23.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:21:23.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:23.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:24.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:25.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:25.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:25.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:21:25.548 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:21:25.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:25.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:26.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:27.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:27.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:27.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:21:27.394 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:21:27.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:27.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:27.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:28.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:28.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:28.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:21:28.923 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:21:28.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:28.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:29.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:30.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:30.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:30.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:21:30.338 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:21:30.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:30.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:31.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:32.024 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:21:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:21:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:32.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:33.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:33.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:33.590 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:21:33.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:21:33.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:33.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:33.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:34.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:34.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:34.964 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:21:34.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:21:34.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:34.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:35.251 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2904983570,v1:127.0.0.1:6827/2904983570] [v2:127.0.0.1:6828/2904983570,v1:127.0.0.1:6829/2904983570] exists,up 0fe1e115-9ef0-4727-a510-86a123c7da5e 2024-03-09T13:21:35.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:21:35.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:21:35.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:21:35.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:807: corrupt_and_repair_jerasure: create_rbd_pool 2024-03-09T13:21:35.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T13:21:35.516 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T13:21:35.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T13:21:35.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T13:21:35.891 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T13:21:35.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:21:36.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T13:21:39.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: corrupt_and_repair_jerasure: wait_for_clean 2024-03-09T13:21:39.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:21:39.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:21:39.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:21:39.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:21:39.453 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:21:39.453 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:21:39.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:21:39.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:21:39.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:21:39.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:21:39.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:21:39.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:21:39.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:21:39.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:21:39.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:21:39.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:21:39.822 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:21:39.822 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:21:39.822 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-03-09T13:21:39.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:21:39.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:21:39.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:21:39.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-03-09T13:21:39.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-03-09T13:21:39.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-03-09T13:21:39.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:21:39.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:21:40.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672978 2024-03-09T13:21:40.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672978 2024-03-09T13:21:40.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672978' 2024-03-09T13:21:40.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:21:40.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:21:40.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509453 2024-03-09T13:21:40.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509453 2024-03-09T13:21:40.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672978 2-64424509453' 2024-03-09T13:21:40.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:21:40.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:21:40.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345925 2024-03-09T13:21:40.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345925 2024-03-09T13:21:40.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-42949672978 2-64424509453 3-85899345925' 2024-03-09T13:21:40.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:21:40.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-03-09T13:21:40.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:21:40.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:21:40.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-03-09T13:21:40.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:21:40.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-03-09T13:21:40.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-03-09T13:21:40.265 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836505 2024-03-09T13:21:40.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:21:40.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-03-09T13:21:40.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:21:41.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:21:41.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:21:41.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-03-09T13:21:41.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:21:42.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:21:42.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:21:43.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836505 2024-03-09T13:21:43.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:21:43.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672978 2024-03-09T13:21:43.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:21:43.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:21:43.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672978 2024-03-09T13:21:43.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:21:43.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672978 2024-03-09T13:21:43.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-03-09T13:21:43.108 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672978 2024-03-09T13:21:43.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:21:43.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672978 2024-03-09T13:21:43.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:21:43.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509453 2024-03-09T13:21:43.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:21:43.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:21:43.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509453 2024-03-09T13:21:43.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:21:43.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509453 2024-03-09T13:21:43.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-03-09T13:21:43.385 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509453 2024-03-09T13:21:43.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:21:43.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509454 -lt 64424509453 2024-03-09T13:21:43.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:21:43.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345925 2024-03-09T13:21:43.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:21:43.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:21:43.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345925 2024-03-09T13:21:43.674 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:21:43.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345925 2024-03-09T13:21:43.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345925' 2024-03-09T13:21:43.676 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345925 2024-03-09T13:21:43.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:21:43.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345925 -lt 85899345925 2024-03-09T13:21:43.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:21:43.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:21:43.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:21:44.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T13:21:44.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:21:44.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:21:44.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:21:44.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:21:44.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:21:44.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:21:44.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:21:44.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:21:44.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:21:44.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:21:44.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:21:44.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T13:21:44.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:21:44.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:21:44.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:810: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2024-03-09T13:21:44.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T13:21:44.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T13:21:44.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-09T13:21:44.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T13:21:44.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2024-03-09T13:21:45.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T13:21:45.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T13:21:45.934 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' created 2024-03-09T13:21:45.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:21:46.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-09T13:21:46.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-09T13:21:47.315 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 2 allow_ec_overwrites to true 2024-03-09T13:21:47.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T13:21:47.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:21:47.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:21:47.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:21:47.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:21:47.328 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:21:47.328 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:21:47.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:21:47.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:21:47.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:21:47.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:21:47.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:21:47.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:21:47.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:21:47.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:21:47.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:21:47.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:21:47.756 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:21:47.756 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:21:47.756 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-03-09T13:21:47.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:21:47.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:21:47.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:21:47.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-03-09T13:21:47.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-03-09T13:21:47.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-03-09T13:21:47.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:21:47.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:21:48.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672981 2024-03-09T13:21:48.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672981 2024-03-09T13:21:48.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672981' 2024-03-09T13:21:48.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:21:48.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:21:48.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509456 2024-03-09T13:21:48.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509456 2024-03-09T13:21:48.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672981 2-64424509456' 2024-03-09T13:21:48.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:21:48.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:21:48.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345927 2024-03-09T13:21:48.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345927 2024-03-09T13:21:48.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-42949672981 2-64424509456 3-85899345927' 2024-03-09T13:21:48.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:21:48.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-03-09T13:21:48.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:21:48.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:21:48.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-03-09T13:21:48.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:21:48.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-03-09T13:21:48.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-03-09T13:21:48.331 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836507 2024-03-09T13:21:48.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:21:48.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-03-09T13:21:48.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:21:49.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:21:49.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:21:49.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-03-09T13:21:49.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:21:50.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:21:50.895 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:21:51.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836508 -lt 21474836507 2024-03-09T13:21:51.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:21:51.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672981 2024-03-09T13:21:51.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:21:51.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:21:51.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672981 2024-03-09T13:21:51.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:21:51.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672981 2024-03-09T13:21:51.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-03-09T13:21:51.182 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672981 2024-03-09T13:21:51.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:21:51.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672981 -lt 42949672981 2024-03-09T13:21:51.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:21:51.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509456 2024-03-09T13:21:51.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:21:51.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:21:51.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509456 2024-03-09T13:21:51.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:21:51.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509456 2024-03-09T13:21:51.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-03-09T13:21:51.459 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509456 2024-03-09T13:21:51.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:21:51.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509456 -lt 64424509456 2024-03-09T13:21:51.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:21:51.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345927 2024-03-09T13:21:51.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:21:51.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:21:51.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345927 2024-03-09T13:21:51.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:21:51.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345927 2024-03-09T13:21:51.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2024-03-09T13:21:51.740 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345927 2024-03-09T13:21:51.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:21:52.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345928 -lt 85899345927 2024-03-09T13:21:52.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:21:52.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:21:52.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:21:52.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:21:52.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:21:52.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:21:52.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:21:52.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:21:52.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:21:52.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:21:52.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:21:52.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:21:52.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:21:52.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:21:52.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:21:52.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:21:52.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:21:52.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:21:52.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T13:21:52.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:811: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-09T13:21:52.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T13:21:52.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-09T13:21:52.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-09T13:21:52.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:21:52.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:21:52.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-09T13:21:52.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:21:52.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:21:52.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:21:53.389 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:21:53.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:21:53.909 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:21:53.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:21:53.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:21:53.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T13:21:54.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-09T13:21:54.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T13:21:54.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:21:54.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:21:54.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:21:54.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-09T13:21:54.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-09T13:21:54.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T13:21:54.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-09T13:21:54.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-09T13:21:54.598 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:21:54.598 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T13:21:54.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-09T13:21:54.894 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:21:54.894 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:21:54.894 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T13:21:54.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 0 2024-03-09T13:21:54.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2024-03-09T13:21:54.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-09T13:21:54.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=1 2024-03-09T13:21:54.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=2 2024-03-09T13:21:54.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-09T13:21:54.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T13:21:54.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T13:21:54.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-09T13:21:54.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:21:54.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:21:54.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:21:54.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:21:54.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:21:54.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:21:54.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:21:54.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:21:54.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:21:54.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:21:54.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:21:54.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:21:54.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:21:54.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:21:54.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:21:54.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:21:55.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:21:55.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:21:55.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:21:55.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:21:55.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:21:55.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:21:55.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:21:55.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T13:21:56.746 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T13:21:57.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:21:57.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:21:57.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:21:57.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:21:57.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:21:57.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:21:57.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:21:57.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:21:57.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:21:57.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:21:57.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:21:57.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:21:57.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:21:57.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:21:57.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:21:57.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:21:57.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:21:57.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:21:57.265 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:21:57.265 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:21:57.265 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:21:57.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:21:57.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:21:57.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:21:57.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:21:57.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:21:57.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:21:57.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:21:57.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:21:57.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:21:57.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:21:57.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:21:57.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:21:57.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:21:57.267 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-03-09T13:21:57.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:21:57.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:21:57.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:21:57.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:21:57.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:21:57.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:21:57.292 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:57.293+0000 7f71badac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:21:57.294 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:57.295+0000 7f71badac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:21:57.296 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:57.296+0000 7f71badac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:21:57.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:21:57.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:21:57.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:21:57.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:21:57.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:21:57.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:57.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:21:57.548 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:21:57.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:57.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:57.817 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:21:57.817+0000 7f71badac640 -1 Falling back to public interface 2024-03-09T13:21:57.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:21:58.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:21:58.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:21:58.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:21:58.825 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:21:58.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:21:58.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:21:59.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:00.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:00.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:00.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:22:00.109 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:22:00.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:00.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:22:00.303 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:00.304+0000 7f71badac640 -1 osd.3 39 log_to_monitors true 2024-03-09T13:22:00.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:01.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:01.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:01.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:22:01.404 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:22:01.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:01.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:22:01.740 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [20,39) [v2:127.0.0.1:6826/3265501522,v1:127.0.0.1:6827/3265501522] [v2:127.0.0.1:6828/3265501522,v1:127.0.0.1:6829/3265501522] exists,up 0fe1e115-9ef0-4727-a510-86a123c7da5e 2024-03-09T13:22:01.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:22:01.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:22:01.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:22:01.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:22:01.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:22:01.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:22:01.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:22:01.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:22:01.741 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:22:01.741 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:22:01.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:22:01.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:22:01.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:22:01.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:22:01.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:22:01.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:22:01.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:22:01.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:22:01.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:22:02.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:22:02.155 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:22:02.155 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:22:02.155 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-03-09T13:22:02.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:22:02.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:02.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:22:02.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836511 2024-03-09T13:22:02.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836511 2024-03-09T13:22:02.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511' 2024-03-09T13:22:02.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:02.272 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:22:02.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672985 2024-03-09T13:22:02.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672985 2024-03-09T13:22:02.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672985' 2024-03-09T13:22:02.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:02.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:22:02.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509460 2024-03-09T13:22:02.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509460 2024-03-09T13:22:02.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672985 2-64424509460' 2024-03-09T13:22:02.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:02.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:22:02.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593730 2024-03-09T13:22:02.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593730 2024-03-09T13:22:02.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836511 1-42949672985 2-64424509460 3-184683593730' 2024-03-09T13:22:02.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:02.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836511 2024-03-09T13:22:02.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:02.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:22:02.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836511 2024-03-09T13:22:02.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:02.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836511 2024-03-09T13:22:02.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-03-09T13:22:02.605 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836511 2024-03-09T13:22:02.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:22:02.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836510 -lt 21474836511 2024-03-09T13:22:02.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:22:03.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:22:03.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:22:04.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836512 -lt 21474836511 2024-03-09T13:22:04.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:04.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672985 2024-03-09T13:22:04.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:04.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:22:04.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672985 2024-03-09T13:22:04.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:04.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672985 2024-03-09T13:22:04.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-03-09T13:22:04.199 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672985 2024-03-09T13:22:04.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:22:04.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672985 -lt 42949672985 2024-03-09T13:22:04.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:04.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509460 2024-03-09T13:22:04.506 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:04.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:22:04.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509460 2024-03-09T13:22:04.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:04.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509460 2024-03-09T13:22:04.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-03-09T13:22:04.508 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509460 2024-03-09T13:22:04.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:22:04.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509460 -lt 64424509460 2024-03-09T13:22:04.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:04.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-184683593730 2024-03-09T13:22:04.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:04.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:22:04.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-184683593730 2024-03-09T13:22:04.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:04.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593730 2024-03-09T13:22:04.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2024-03-09T13:22:04.787 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 184683593730 2024-03-09T13:22:04.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:22:05.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593730 -lt 184683593730 2024-03-09T13:22:05.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:22:05.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:22:05.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:22:05.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:22:05.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:22:05.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:22:05.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:22:05.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:22:05.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:22:05.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:22:05.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:22:05.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:22:05.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:22:05.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:22:05.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:22:06.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:22:06.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:22:06.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:22:06.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T13:22:06.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:22:06.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:22:06.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:22:06.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:22:06.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T13:22:06.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T13:22:06.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:22:06.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:22:06.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:22:06.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:22:06.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:22:06.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:22:06.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:21:45.767793+0000 2024-03-09T13:22:06.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:22:06.829 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:22:06.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:21:45.767793+0000 2024-03-09T13:22:06.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:22:06.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:21:45.767793+0000 2024-03-09T13:22:06.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:22:06.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:22:06.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:22:06.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:22:06.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:22:06.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:22:06.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:22:06.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:22:07.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:21:45.767793+0000 '>' 2024-03-09T13:21:45.767793+0000 2024-03-09T13:22:07.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:22:08.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:22:08.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:22:08.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:22:08.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:22:08.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:22:08.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:22:08.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:22:08.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:21:45.767793+0000 '>' 2024-03-09T13:21:45.767793+0000 2024-03-09T13:22:08.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:22:09.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:22:09.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:22:09.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:22:09.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:22:09.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:22:09.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:22:09.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:22:09.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:21:45.767793+0000 '>' 2024-03-09T13:21:45.767793+0000 2024-03-09T13:22:09.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:22:10.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:22:10.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:22:10.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:22:10.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:22:10.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:22:10.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:22:10.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:22:10.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:21:45.767793+0000 '>' 2024-03-09T13:21:45.767793+0000 2024-03-09T13:22:10.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:22:11.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:22:11.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:22:11.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:22:11.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:22:11.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:22:11.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:22:11.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:22:12.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:22:07.228588+0000 '>' 2024-03-09T13:21:45.767793+0000 2024-03-09T13:22:12.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:22:12.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:22:12.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:22:12.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:22:12.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:22:12.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:22:12.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:22:12.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:22:12.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:22:12.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:22:12.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:22:12.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:22:12.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:22:12.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:22:12.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:22:12.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:22:12.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:22:12.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:22:12.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:22:12.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:22:12.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:22:12.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:22:12.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:22:12.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:22:12.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T13:22:14.255 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T13:22:14.256 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:22:14.256 INFO:tasks.workunit.client.0.smithi162.stdout:hinfo_key 2024-03-09T13:22:14.256 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:22:14.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:22:14.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:22:14.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:22:14.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:22:14.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:22:14.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:22:14.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:22:14.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:22:14.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:22:14.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:22:14.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:22:14.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:22:14.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:22:14.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:22:14.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:22:14.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:22:14.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:22:14.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:22:14.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:22:14.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:22:14.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:22:14.526 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-03-09T13:22:14.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:22:14.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:22:14.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:22:14.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:22:14.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:22:14.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:22:14.553 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:14.553+0000 7f5abecd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:14.555 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:14.556+0000 7f5abecd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:14.556 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:14.557+0000 7f5abecd3640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:22:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:22:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:22:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:22:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:22:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:22:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:22:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:14.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:22:15.072 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:15.072+0000 7f5abecd3640 -1 Falling back to public interface 2024-03-09T13:22:15.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:16.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:16.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:16.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:22:16.084 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:22:16.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:16.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:22:16.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:17.326 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:17.327+0000 7f5abecd3640 -1 osd.3 45 log_to_monitors true 2024-03-09T13:22:17.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:17.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:17.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:22:17.364 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:22:17.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:17.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:22:17.824 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 50 up_thru 43 down_at 46 last_clean_interval [43,45) [v2:127.0.0.1:6826/3589762687,v1:127.0.0.1:6827/3589762687] [v2:127.0.0.1:6828/3589762687,v1:127.0.0.1:6829/3589762687] exists,up 0fe1e115-9ef0-4727-a510-86a123c7da5e 2024-03-09T13:22:17.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:22:17.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:22:17.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:22:17.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:22:17.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:22:17.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:22:17.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:22:17.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:22:17.825 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:22:17.825 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:22:17.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:22:17.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:22:17.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:22:17.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:22:17.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:22:17.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:22:17.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:22:17.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:22:17.911 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:22:18.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:22:18.400 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:22:18.400 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:22:18.400 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-03-09T13:22:18.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:22:18.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:18.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:22:18.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836516 2024-03-09T13:22:18.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836516 2024-03-09T13:22:18.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516' 2024-03-09T13:22:18.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:18.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:22:18.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672989 2024-03-09T13:22:18.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672989 2024-03-09T13:22:18.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672989' 2024-03-09T13:22:18.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:18.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:22:18.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509464 2024-03-09T13:22:18.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509464 2024-03-09T13:22:18.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672989 2-64424509464' 2024-03-09T13:22:18.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:18.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:22:19.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364802 2024-03-09T13:22:19.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364802 2024-03-09T13:22:19.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836516 1-42949672989 2-64424509464 3-214748364802' 2024-03-09T13:22:19.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:19.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836516 2024-03-09T13:22:19.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:19.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:22:19.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836516 2024-03-09T13:22:19.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:19.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836516 2024-03-09T13:22:19.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-03-09T13:22:19.019 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836516 2024-03-09T13:22:19.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:22:19.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836514 -lt 21474836516 2024-03-09T13:22:19.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:22:20.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:22:20.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:22:20.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836516 -lt 21474836516 2024-03-09T13:22:20.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:20.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672989 2024-03-09T13:22:20.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:20.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:22:20.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672989 2024-03-09T13:22:20.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:20.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672989 2024-03-09T13:22:20.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-03-09T13:22:20.685 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672989 2024-03-09T13:22:20.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:22:20.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672989 -lt 42949672989 2024-03-09T13:22:20.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:20.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509464 2024-03-09T13:22:20.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:20.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:22:20.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509464 2024-03-09T13:22:20.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:20.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509464 2024-03-09T13:22:20.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-03-09T13:22:20.963 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509464 2024-03-09T13:22:20.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:22:21.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509464 -lt 64424509464 2024-03-09T13:22:21.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:21.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-214748364802 2024-03-09T13:22:21.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:21.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:22:21.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-214748364802 2024-03-09T13:22:21.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:21.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364802 2024-03-09T13:22:21.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 214748364802' 2024-03-09T13:22:21.240 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 214748364802 2024-03-09T13:22:21.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:22:21.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364802 -lt 214748364802 2024-03-09T13:22:21.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:22:21.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:22:21.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:22:21.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:22:21.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:22:21.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:22:21.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:22:21.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:22:21.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:22:21.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:22:21.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:22:22.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:22:22.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:22:22.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:22:22.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:22:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:22:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T13:22:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:22:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T13:22:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:22:22.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:22:22.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:22:22.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:22:22.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:22:22.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:22:22.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:22:22.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:22:22.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:22:22.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:22:22.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:22:22.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:22:22.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:22:22.914 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:22:23.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:22:23.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:22:23.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:22:23.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:22:23.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:22:23.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:22:23.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:22:23.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:22:23.254 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:22:23.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:22:23.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:22:23.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:22:23.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:22:23.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:22:23.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:22:23.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:22:23.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:22:23.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:22:23.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:22:23.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:22:23.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:22:23.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:22:24.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:22:24.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:22:24.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:22:24.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:22:24.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:22:24.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:22:24.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:22:24.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:22:24.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:22:24.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2024-03-09T13:22:24.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T13:22:24.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T13:22:24.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-09T13:22:24.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:22:24.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:22:24.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:22:24.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:22:24.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:22:24.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:22:24.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:22:24.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:22:24.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:22:24.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:22:24.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:22:24.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:22:24.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:22:24.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:22:24.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:22:24.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:22:24.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:22:24.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:22:24.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:22:24.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:22:24.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:22:24.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:22:24.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:22:24.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T13:22:26.623 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:22:27.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:22:27.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:22:27.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:22:27.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:22:27.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:22:27.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:22:27.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:22:27.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:22:27.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:22:27.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:22:27.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:22:27.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:22:27.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:22:27.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:22:27.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:22:27.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:22:27.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:22:27.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:22:27.142 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:22:27.142 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:22:27.142 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:22:27.142 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:22:27.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:22:27.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:22:27.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:22:27.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:22:27.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:22:27.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:22:27.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:22:27.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:22:27.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:22:27.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:22:27.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:22:27.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:22:27.144 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:22:27.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:22:27.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:22:27.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:22:27.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:22:27.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:22:27.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:22:27.172 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:27.174+0000 7f36a6bf8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:27.174 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:27.176+0000 7f36a6bf8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:27.176 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:27.177+0000 7f36a6bf8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:27.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:22:27.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:22:27.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:22:27.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:22:27.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:22:27.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:27.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:22:27.426 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:22:27.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:27.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:22:27.690 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:27.692+0000 7f36a6bf8640 -1 Falling back to public interface 2024-03-09T13:22:27.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:28.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:28.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:28.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:22:28.703 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:22:28.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:28.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:22:28.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:29.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:29.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:29.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:22:29.986 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:22:29.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:29.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:22:30.336 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:30.338+0000 7f36a6bf8640 -1 osd.1 52 log_to_monitors true 2024-03-09T13:22:30.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:31.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:31.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:31.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:22:31.354 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:22:31.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:31.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:22:31.768 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [10,52) [v2:127.0.0.1:6810/1261476023,v1:127.0.0.1:6811/1261476023] [v2:127.0.0.1:6812/1261476023,v1:127.0.0.1:6813/1261476023] exists,up 9dbfada3-cce6-4508-b697-011f9f8cf024 2024-03-09T13:22:31.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:22:31.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:22:31.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:22:31.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:22:31.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:22:31.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:22:31.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:22:31.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:22:31.769 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:22:31.769 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:22:31.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:22:31.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:22:31.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:22:31.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:22:31.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:22:31.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:22:31.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:22:31.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:22:31.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:22:32.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:22:32.177 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:22:32.177 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:22:32.177 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-03-09T13:22:32.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:22:32.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:32.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:22:32.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836519 2024-03-09T13:22:32.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836519 2024-03-09T13:22:32.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519' 2024-03-09T13:22:32.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:32.312 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:22:32.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135874 2024-03-09T13:22:32.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135874 2024-03-09T13:22:32.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-244813135874' 2024-03-09T13:22:32.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:32.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:22:32.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509468 2024-03-09T13:22:32.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509468 2024-03-09T13:22:32.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-244813135874 2-64424509468' 2024-03-09T13:22:32.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:32.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:22:32.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364806 2024-03-09T13:22:32.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364806 2024-03-09T13:22:32.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836519 1-244813135874 2-64424509468 3-214748364806' 2024-03-09T13:22:32.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:32.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836519 2024-03-09T13:22:32.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:32.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:22:32.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836519 2024-03-09T13:22:32.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:32.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836519 2024-03-09T13:22:32.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2024-03-09T13:22:32.657 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836519 2024-03-09T13:22:32.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:22:32.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836518 -lt 21474836519 2024-03-09T13:22:32.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:22:33.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:22:33.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:22:34.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836520 -lt 21474836519 2024-03-09T13:22:34.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:34.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135874 2024-03-09T13:22:34.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:34.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:22:34.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135874 2024-03-09T13:22:34.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:34.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135874 2024-03-09T13:22:34.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-03-09T13:22:34.236 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 244813135874 2024-03-09T13:22:34.236 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:22:34.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135874 -lt 244813135874 2024-03-09T13:22:34.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:34.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509468 2024-03-09T13:22:34.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:34.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:22:34.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509468 2024-03-09T13:22:34.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:34.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509468 2024-03-09T13:22:34.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-03-09T13:22:34.514 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509468 2024-03-09T13:22:34.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:22:34.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509468 -lt 64424509468 2024-03-09T13:22:34.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:34.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-214748364806 2024-03-09T13:22:34.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:34.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:22:34.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-214748364806 2024-03-09T13:22:34.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:34.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364806 2024-03-09T13:22:34.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 214748364806' 2024-03-09T13:22:34.792 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 214748364806 2024-03-09T13:22:34.792 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:22:35.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364806 -lt 214748364806 2024-03-09T13:22:35.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:22:35.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:22:35.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:22:35.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:22:35.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:22:35.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:22:35.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:22:35.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:22:35.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:22:35.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:22:35.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:22:35.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:22:35.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:22:35.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:22:35.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:22:36.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:22:36.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:22:36.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:22:36.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T13:22:36.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:22:36.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:22:36.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:22:36.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:22:36.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T13:22:36.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T13:22:36.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:22:36.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:22:36.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:22:36.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:22:36.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:22:36.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:22:36.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:22:07.228588+0000 2024-03-09T13:22:36.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:22:36.825 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:22:36.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:22:07.228588+0000 2024-03-09T13:22:36.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:22:36.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:22:07.228588+0000 2024-03-09T13:22:36.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:22:36.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:22:36.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:22:36.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:22:36.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:22:36.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:22:36.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:22:36.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:22:37.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:22:07.228588+0000 '>' 2024-03-09T13:22:07.228588+0000 2024-03-09T13:22:37.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:22:38.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:22:38.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:22:38.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:22:38.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:22:38.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:22:38.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:22:38.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:22:38.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:22:37.404772+0000 '>' 2024-03-09T13:22:07.228588+0000 2024-03-09T13:22:38.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:22:38.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:22:38.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:22:38.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:22:38.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:22:38.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:22:38.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:22:38.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:22:38.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:22:38.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:22:38.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:22:38.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:22:38.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:22:38.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:22:38.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:22:38.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:22:38.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:22:38.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:22:38.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:22:38.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:22:38.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:22:38.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:22:38.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:22:38.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:22:38.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T13:22:40.408 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T13:22:40.408 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:22:40.408 INFO:tasks.workunit.client.0.smithi162.stdout:hinfo_key 2024-03-09T13:22:40.408 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:22:40.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:22:40.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:22:40.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:22:40.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:22:40.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:22:40.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:22:40.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:22:40.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:22:40.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:22:40.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:22:40.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:22:40.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:22:40.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:22:40.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:22:40.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:22:40.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:22:40.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:22:40.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:22:40.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:22:40.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:22:40.682 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-03-09T13:22:40.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:22:40.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:22:40.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:22:40.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:22:40.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:22:40.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:22:40.709 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:40.710+0000 7f5ad6bac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:40.711 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:40.713+0000 7f5ad6bac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:40.713 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:40.714+0000 7f5ad6bac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:40.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:22:40.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:22:40.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:22:40.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:22:40.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:22:40.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:40.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:22:40.966 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:22:40.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:40.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:22:41.232 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:41.233+0000 7f5ad6bac640 -1 Falling back to public interface 2024-03-09T13:22:41.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:42.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:42.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:42.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:22:42.239 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:22:42.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:42.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:22:42.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:43.416 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:22:43.416+0000 7f5ad6bac640 -1 osd.3 59 log_to_monitors true 2024-03-09T13:22:43.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:43.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:43.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:22:43.524 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:22:43.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:43.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:22:43.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:44.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:44.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:44.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:22:44.813 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:22:44.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:44.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:22:45.285 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 64 up_thru 64 down_at 60 last_clean_interval [50,59) [v2:127.0.0.1:6826/4162551248,v1:127.0.0.1:6827/4162551248] [v2:127.0.0.1:6828/4162551248,v1:127.0.0.1:6829/4162551248] exists,up 0fe1e115-9ef0-4727-a510-86a123c7da5e 2024-03-09T13:22:45.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:22:45.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:22:45.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:22:45.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:22:45.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:22:45.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:22:45.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:22:45.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:22:45.286 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:22:45.286 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:22:45.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:22:45.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:22:45.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:22:45.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:22:45.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:22:45.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:22:45.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:22:45.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:22:45.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:22:45.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:22:45.670 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:22:45.670 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:22:45.670 INFO:tasks.workunit.client.0.smithi162.stderr:3' 2024-03-09T13:22:45.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:22:45.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:45.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:22:45.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836523 2024-03-09T13:22:45.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836523 2024-03-09T13:22:45.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523' 2024-03-09T13:22:45.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:45.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:22:45.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135878 2024-03-09T13:22:45.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135878 2024-03-09T13:22:45.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-244813135878' 2024-03-09T13:22:45.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:45.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:22:46.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509471 2024-03-09T13:22:46.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509471 2024-03-09T13:22:46.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-244813135878 2-64424509471' 2024-03-09T13:22:46.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:46.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:22:46.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=274877906946 2024-03-09T13:22:46.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 274877906946 2024-03-09T13:22:46.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836523 1-244813135878 2-64424509471 3-274877906946' 2024-03-09T13:22:46.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:46.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836523 2024-03-09T13:22:46.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:46.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:22:46.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836523 2024-03-09T13:22:46.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:46.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836523 2024-03-09T13:22:46.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-03-09T13:22:46.132 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836523 2024-03-09T13:22:46.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:22:46.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836523 -lt 21474836523 2024-03-09T13:22:46.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:46.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135878 2024-03-09T13:22:46.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:46.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:22:46.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135878 2024-03-09T13:22:46.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:46.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135878 2024-03-09T13:22:46.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135878' 2024-03-09T13:22:46.406 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 244813135878 2024-03-09T13:22:46.407 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:22:46.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135877 -lt 244813135878 2024-03-09T13:22:46.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:22:47.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:22:47.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:22:47.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135878 -lt 244813135878 2024-03-09T13:22:47.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:47.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509471 2024-03-09T13:22:47.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:47.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:22:47.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509471 2024-03-09T13:22:47.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:47.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509471 2024-03-09T13:22:47.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-03-09T13:22:47.972 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509471 2024-03-09T13:22:47.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:22:48.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509472 -lt 64424509471 2024-03-09T13:22:48.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:22:48.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-274877906946 2024-03-09T13:22:48.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:22:48.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:22:48.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-274877906946 2024-03-09T13:22:48.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:22:48.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=274877906946 2024-03-09T13:22:48.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 274877906946' 2024-03-09T13:22:48.246 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 274877906946 2024-03-09T13:22:48.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:22:48.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 274877906946 -lt 274877906946 2024-03-09T13:22:48.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:22:48.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:22:48.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:22:48.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:22:48.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:22:48.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:22:48.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:22:48.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:22:48.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:22:48.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:22:48.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:22:49.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:22:49.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:22:49.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:22:49.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:22:49.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:22:49.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:22:49.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:22:49.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:22:49.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:22:49.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2024-03-09T13:22:49.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T13:22:49.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T13:22:49.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=1 2024-03-09T13:22:49.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=2 2024-03-09T13:22:49.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T13:22:49.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:22:49.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191779"' 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191779' 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 191781"' 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 191781' 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 191779 191781' 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191779 2024-03-09T13:22:49.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191782: /' 2024-03-09T13:22:49.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T13:22:49.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/191784: /' 2024-03-09T13:22:52.387 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:22:52.387 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:22:52.388 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:22:52.388 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:22:52.388 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:22:52.388 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:22:52.388 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:22:52.388 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:22:52.388 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:22:52.388 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:22:52.388 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:22:52.388 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:22:52.388 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:22:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:22:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:22:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:22:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:22:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:22:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:22:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:22:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:22:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:22:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T13:22:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:191782: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:22:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:22:52.389 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:22:52.390 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:22:52.390 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:22:52.390 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:22:52.390 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:22:52.390 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:22:52.390 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:22:52.391 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:22:52.391 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:22:52.391 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:22:52.391 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:22:52.391 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:22:52.391 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:22:52.391 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:22:52.391 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:22:52.391 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:22:52.391 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:22:52.391 INFO:tasks.workunit.client.0.smithi162.stderr:191782: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:22:52.392 INFO:tasks.workunit.client.0.smithi162.stderr:191782: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:22:52.392 INFO:tasks.workunit.client.0.smithi162.stderr:191782: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:22:52.392 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:22:52.392 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:22:52.392 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:22:52.392 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:22:52.392 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:22:52.392 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:22:52.392 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:22:52.392 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:22:52.392 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:22:52.392 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:22:52.393 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:22:52.393 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:22:52.393 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:22:52.393 INFO:tasks.workunit.client.0.smithi162.stderr:191782: start osd.1 2024-03-09T13:22:52.393 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:22:52.454 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:22:52.455 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:22:52.455 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T13:22:52.455 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:22:52.455 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T13:22:52.455 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:22:52.455 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:22:52.455 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T13:22:52.455 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:22:52.455 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T13:22:52.455 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:22:52.455 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:22:52.455 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:22:52.456 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T13:22:52.457 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:22:52.457 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:22:52.457 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:22:52.457 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:22:52.457 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:22:52.457 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:22:52.457 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:22:52.457 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:22:52.457 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:22:52.458 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:22:52.458 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:22:52.458 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:22:52.458 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:22:52.458 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:22:52.458 INFO:tasks.workunit.client.0.smithi162.stderr:191784: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:22:52.458 INFO:tasks.workunit.client.0.smithi162.stderr:191784: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:22:52.458 INFO:tasks.workunit.client.0.smithi162.stderr:191784: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:22:52.458 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:22:52.458 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:22:52.458 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:22:52.458 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:22:52.459 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:22:52.459 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:22:52.459 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:22:52.459 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:22:52.459 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:22:52.459 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:22:52.459 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:22:52.459 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:22:52.459 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T13:22:52.459 INFO:tasks.workunit.client.0.smithi162.stderr:191784: start osd.2 2024-03-09T13:22:52.459 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:22:57.025 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:22:57.025 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:22:57.025 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:22:57.025 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:22:57.025 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:22:57.025 INFO:tasks.workunit.client.0.smithi162.stderr:191782: 2024-03-09T13:22:52.420+0000 7fcdc7213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:57.025 INFO:tasks.workunit.client.0.smithi162.stderr:191782: 2024-03-09T13:22:52.422+0000 7fcdc7213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:57.025 INFO:tasks.workunit.client.0.smithi162.stderr:191782: 2024-03-09T13:22:52.423+0000 7fcdc7213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:57.025 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:22:57.025 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:22:57.026 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:22:57.026 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:22:57.026 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:22:57.026 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:57.026 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:22:57.026 INFO:tasks.workunit.client.0.smithi162.stderr:191782: 0 2024-03-09T13:22:57.026 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:57.026 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:22:57.026 INFO:tasks.workunit.client.0.smithi162.stderr:191782: 2024-03-09T13:22:52.947+0000 7fcdc7213640 -1 Falling back to public interface 2024-03-09T13:22:57.026 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: 1 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: 2 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:22:57.027 INFO:tasks.workunit.client.0.smithi162.stderr:191782: 2024-03-09T13:22:55.556+0000 7fcdc7213640 -1 osd.1 66 log_to_monitors true 2024-03-09T13:22:57.028 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:57.028 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:57.028 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:57.028 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:22:57.028 INFO:tasks.workunit.client.0.smithi162.stderr:191782: 3 2024-03-09T13:22:57.028 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:57.028 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:22:57.222 INFO:tasks.workunit.client.0.smithi162.stderr:191782: osd.1 up in weight 1 up_from 72 up_thru 72 down_at 67 last_191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T13:22:57.222 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T13:22:57.222 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:22:57.223 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:22:57.223 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:22:57.223 INFO:tasks.workunit.client.0.smithi162.stderr:191784: 2024-03-09T13:22:52.487+0000 7f364460e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:57.223 INFO:tasks.workunit.client.0.smithi162.stderr:191784: 2024-03-09T13:22:52.489+0000 7f364460e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:57.223 INFO:tasks.workunit.client.0.smithi162.stderr:191784: 2024-03-09T13:22:52.490+0000 7f364460e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:22:57.223 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T13:22:57.223 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:22:57.223 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:22:57.223 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:22:57.223 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:22:57.223 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:57.223 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:22:57.223 INFO:tasks.workunit.client.0.smithi162.stderr:191784: 0 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: 2024-03-09T13:22:53.020+0000 7f364460e640 -1 Falling back to public interface 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: 1 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:57.224 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:22:57.225 INFO:tasks.workunit.client.0.smithi162.stderr:191784: 2 2024-03-09T13:22:57.225 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:57.225 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:22:57.225 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:22:57.225 INFO:tasks.workunit.client.0.smithi162.stderr:191784: 2024-03-09T13:22:55.905+0000 7f364460e640 -1 osd.2 66 log_to_monitors true 2024-03-09T13:22:57.225 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:22:57.225 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:22:57.225 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:22:57.225 INFO:tasks.workunit.client.0.smithi162.stderr:191784: 3 2024-03-09T13:22:57.225 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:22:57.225 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:22:57.840 INFO:tasks.workunit.client.0.smithi162.stderr:191784: osd.2 up in weight 1 up_from 73 up_thru 53 down_at 67 last_clean_interval [57,66) [v2:127.0.0.1:6810/2115080501,v1:127.0.0.1:6811/2115080501] [v2:127.0.0.1:6812/2115080501,v1:127.0.0.1:6813/2115080501] exists,up 9dbfada3-cce6-4508-b697-011f9f8cf024 2024-03-09T13:22:57.840 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:22:57.840 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:22:57.840 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:22:57.840 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:22:57.840 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:22:57.840 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:22:57.841 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: 1 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: 2 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: 3' 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836526 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836526 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526' 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-03-09T13:22:57.842 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-309237645314' 2024-03-09T13:22:57.843 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph clean_interval [15,66) [v2:127.0.0.1:6818/117620137,v1:127.0.0.1:6819/117620137] [v2:127.0.0.1:6820/117620137,v1:127.0.0.1:6821/117620137] exists,up 5c7f558e-01b7-4c13-abba-dec214f5e7f0 2024-03-09T13:22:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:22:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:22:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:22:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:22:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:22:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:22:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:22:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: 1 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: 2 2024-03-09T13:22:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:191784: 3' 2024-03-09T13:22:57.961 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:22:57.961 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:57.961 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:22:57.961 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836527 2024-03-09T13:22:57.961 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836527 2024-03-09T13:22:57.961 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527' 2024-03-09T13:22:57.961 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:22:57.961 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:22:57.961 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645315 2024-03-09T13:22:57.961 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645315 2024-03-09T13:22:57.961 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-309237645315' 2024-03-09T13:22:57.961 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:00.062 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph telltell osd.2 flush_pg_stats 2024-03-09T13:23:00.062 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-03-09T13:23:00.062 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-03-09T13:23:00.062 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-309237645314 2-313532612610' 2024-03-09T13:23:00.062 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:00.062 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:23:00.062 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=274877906949 2024-03-09T13:23:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 274877906949 2024-03-09T13:23:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836526 1-309237645314 2-313532612610 3-274877906949' 2024-03-09T13:23:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836526 2024-03-09T13:23:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:23:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836526 2024-03-09T13:23:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836526 2024-03-09T13:23:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-03-09T13:23:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:191782: waiting osd.0 seq 21474836526 2024-03-09T13:23:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:00.063 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836526 -lt 21474836526 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-309237645314 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-309237645314 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 309237645314' 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: waiting osd.1 seq 309237645314 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 309237645314 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:23:00.064 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:23:00.065 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645315 -lt 309237645314 2024-03-09T13:23:00.065 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:01.201 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //hom osd.2 flush_pg_stats 2024-03-09T13:23:01.201 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612611 2024-03-09T13:23:01.201 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612611 2024-03-09T13:23:01.201 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-309237645315 2-313532612611' 2024-03-09T13:23:01.201 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:01.201 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=274877906950 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 274877906950 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836527 1-309237645315 2-313532612611 3-274877906950' 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836527 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836527 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836527 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: waiting osd.0 seq 21474836527 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836526 -lt 21474836527 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836526 -lt 21474836527 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836528 -lt 21474836527 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-309237645315 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-309237645315 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645315 2024-03-09T13:23:01.258 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2e/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612610 2024-03-09T13:23:01.258 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612610 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: waiting osd.2 seq 313532612610 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612611 -lt 313532612610 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-274877906949 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-274877906949 2024-03-09T13:23:01.259 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=274877906949 2024-03-09T13:23:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 274877906949' 2024-03-09T13:23:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:191782: waiting osd.3 seq 274877906949 2024-03-09T13:23:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:23:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 274877906951 -lt 274877906949 2024-03-09T13:23:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:23:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:23:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:01.260 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:01.261 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:01.261 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:01.261 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:01.261 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:23:01.261 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:01.647 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:01.648 INFO:tasks.workunit.client.0.smithi162.stderr:191782: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:01.648 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:23:01.648 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:23:01.648 INFO:tasks.workunit.client.0.smithi162.stderr:191782: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:23:01.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:23:01.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:23:01.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 191781 2024-03-09T13:23:02.439 INFO:tasks.workunit.client.0.smithi162.stderr:213: flush_pg_stats: echo 'waiting osd.1 seq 309237645315' 2024-03-09T13:23:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:191784: waiting osd.1 seq 309237645315 2024-03-09T13:23:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:23:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645315 -lt 309237645315 2024-03-09T13:23:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612611 2024-03-09T13:23:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:23:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612611 2024-03-09T13:23:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612611 2024-03-09T13:23:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612611' 2024-03-09T13:23:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:191784: waiting osd.2 seq 313532612611 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612611 -lt 313532612611 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-274877906950 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-274877906950 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=274877906950 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 274877906950' 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: waiting osd.3 seq 274877906950 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 274877906951 -lt 274877906950 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:23:02.441 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:02.442 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:02.442 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:23:02.442 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:02.442 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:02.442 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:02.442 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:02.442 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:02.442 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:03.051 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:03.051 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:23:03.051 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:03.052 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:03.052 INFO:tasks.workunit.client.0.smithi162.stderr:191784: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:03.052 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:23:03.052 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:23:03.052 INFO:tasks.workunit.client.0.smithi162.stderr:191784: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:23:03.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:23:03.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:23:03.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:23:03.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:23:03.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T13:23:03.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:23:03.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T13:23:03.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:23:03.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:23:03.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:23:03.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:23:03.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T13:23:03.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T13:23:03.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:23:03.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:23:03.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:23:03.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:23:03.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:23:03.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:23:03.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:22:37.404772+0000 2024-03-09T13:23:03.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:23:03.893 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:23:03.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:22:37.404772+0000 2024-03-09T13:23:03.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:23:03.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:22:37.404772+0000 2024-03-09T13:23:03.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:23:03.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:23:03.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:23:03.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:23:03.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:23:03.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:23:03.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:23:03.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:23:04.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:22:37.404772+0000 '>' 2024-03-09T13:22:37.404772+0000 2024-03-09T13:23:04.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:23:05.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:23:05.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:23:05.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:23:05.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:23:05.207 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:23:05.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:23:05.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:23:05.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:22:37.404772+0000 '>' 2024-03-09T13:22:37.404772+0000 2024-03-09T13:23:05.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:23:06.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:23:06.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:23:06.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:23:06.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:23:06.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:23:06.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:23:06.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:23:06.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:22:37.404772+0000 '>' 2024-03-09T13:22:37.404772+0000 2024-03-09T13:23:06.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:23:07.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:23:07.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:23:07.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:23:07.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:23:07.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:23:07.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:23:07.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:23:08.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:22:37.404772+0000 '>' 2024-03-09T13:22:37.404772+0000 2024-03-09T13:23:08.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:23:09.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:23:09.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:23:09.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:23:09.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:23:09.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:23:09.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:23:09.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:23:09.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:22:37.404772+0000 '>' 2024-03-09T13:22:37.404772+0000 2024-03-09T13:23:09.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:23:10.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:23:10.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:23:10.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:23:10.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:23:10.324 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:23:10.324 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:23:10.324 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:23:10.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:23:04.465223+0000 '>' 2024-03-09T13:22:37.404772+0000 2024-03-09T13:23:10.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:23:10.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T13:23:10.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:23:10.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:23:10.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:23:10.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 194679"' 2024-03-09T13:23:10.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 194679' 2024-03-09T13:23:10.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T13:23:10.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:23:10.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:23:10.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 194681"' 2024-03-09T13:23:10.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:23:10.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 194681' 2024-03-09T13:23:10.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T13:23:10.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 194679 194681' 2024-03-09T13:23:10.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/194682: /' 2024-03-09T13:23:10.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:23:10.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:23:10.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 194679 2024-03-09T13:23:10.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T13:23:10.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/194684: /' 2024-03-09T13:23:13.033 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:23:13.033 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:23:13.033 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:23:13.033 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:23:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:23:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:23:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:23:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:23:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:23:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:23:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:23:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:23:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:23:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:23:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:23:13.034 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:23:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:23:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:23:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:23:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:23:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:23:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:23:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T13:23:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:194682: _ 2024-03-09T13:23:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:194682: hinfo_key 2024-03-09T13:23:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:194682: snapset 2024-03-09T13:23:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:23:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:23:13.035 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:23:13.036 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:23:13.036 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:23:13.036 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:23:13.036 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:23:13.036 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:23:13.036 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:23:13.036 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:13.037 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:23:13.038 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:23:13.038 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:23:13.038 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:23:13.038 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:23:13.038 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:23:13.038 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:23:13.038 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:23:13.038 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:23:13.038 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:23:13.038 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:23:13.038 INFO:tasks.workunit.client.0.smithi162.stderr:194682: start osd.1 2024-03-09T13:23:13.074 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:23:13.075 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:23:13.075 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T13:23:13.075 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:23:13.075 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T13:23:13.075 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:23:13.075 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:23:13.075 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T13:23:13.075 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:23:13.075 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T13:23:13.075 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:23:13.075 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:23:13.075 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:23:13.075 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:23:13.076 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:23:13.076 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:23:13.076 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2024-03-09T13:23:13.076 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:23:13.076 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:23:13.076 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T13:23:13.076 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:23:13.076 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:23:13.076 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2024-03-09T13:23:13.076 INFO:tasks.workunit.client.0.smithi162.stderr:194684: _ 2024-03-09T13:23:13.077 INFO:tasks.workunit.client.0.smithi162.stderr:194684: hinfo_key 2024-03-09T13:23:13.077 INFO:tasks.workunit.client.0.smithi162.stderr:194684: snapset 2024-03-09T13:23:13.077 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T13:23:13.077 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:23:13.077 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:23:13.077 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T13:23:13.077 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:23:13.077 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:23:13.077 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:23:13.077 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:23:13.078 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:23:13.078 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:23:13.078 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:23:13.078 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:23:13.078 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:23:13.078 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:23:13.078 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:23:13.078 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:23:13.078 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:23:13.079 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:23:13.079 INFO:tasks.workunit.client.0.smithi162.stderr:194684: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:23:13.079 INFO:tasks.workunit.client.0.smithi162.stderr:194684: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:13.079 INFO:tasks.workunit.client.0.smithi162.stderr:194684: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:13.079 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:13.079 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:13.079 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:23:13.079 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:23:13.079 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:23:13.079 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:23:13.079 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:23:13.080 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:23:13.080 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:23:13.080 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:23:13.080 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:23:13.080 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:23:13.080 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T13:23:13.080 INFO:tasks.workunit.client.0.smithi162.stderr:194684: start osd.2 2024-03-09T13:23:17.755 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:23:17.755 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T13:23:17.755 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T13:23:17.755 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:23:17.755 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:23:17.755 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:23:17.755 INFO:tasks.workunit.client.0.smithi162.stderr:194684: 2024-03-09T13:23:13.106+0000 7f5942ed1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:17.755 INFO:tasks.workunit.client.0.smithi162.stderr:194684: 2024-03-09T13:23:13.108+0000 7f5942ed1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:17.755 INFO:tasks.workunit.client.0.smithi162.stderr:194684: 2024-03-09T13:23:13.109+0000 7f5942ed1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: 0 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: 2024-03-09T13:23:13.640+0000 7f5942ed1640 -1 Falling back to public interface 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:17.756 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: 1 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: 2 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: 2024-03-09T13:23:16.326+0000 7f5942ed1640 -1 osd.2 75 log_to_monitors true 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:17.757 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194684: 3 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194684: osd.2 up in weight 1 up_from 81 up_thru 53 down_aovery_ops 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194682: 2024-03-09T13:23:13.066+0000 7f2537c1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194682: 2024-03-09T13:23:13.068+0000 7f2537c1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194682: 2024-03-09T13:23:13.069+0000 7f2537c1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:23:17.758 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: 0 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: 2024-03-09T13:23:13.594+0000 7f2537c1e640 -1 Falling back to public interface 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:23:17.759 INFO:tasks.workunit.client.0.smithi162.stderr:194682: 1 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: 2 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: 2024-03-09T13:23:16.243+0000 7f2537c1e640 -1 osd.1 75 log_to_monitors true 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:23:17.760 INFO:tasks.workunit.client.0.smithi162.stderr:194682: 3 2024-03-09T13:23:17.761 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:17.761 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:23:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:194682: osd.1 up in weight 1 up_from 81 up_thru 73 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/145420859,v1:127.0.0.1:6819/145420859] [v2:127.0.0.1:6820/145420859,v1:127.0.0.1:6821/145420859] exists,up 5c7f558e-01b7-4c13-abba-dec214f5e7f0 2024-03-09T13:23:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:23:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:23:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:23:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:23:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:23:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:23:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:23:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:23:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:194684: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:23:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:194684: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:23:18.453 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: 1 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: 2 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: 3' 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:18.454 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836532 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836532 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532' 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350979 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350979 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-347892350979' 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: t 76 last_clean_interval [72,75) [v2:127.0.0.1:6810/3944520251,v1:127.0.0.1:6811/3944520251] [v2:127.0.0.1:6812/3944520251,v1:127.0.0.1:6813/3944520251] exists,up 9dbfada3-cce6-4508-b697-011f9f8cf024 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:23:18.455 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:23:18.456 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:23:18.456 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:23:18.456 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:23:18.456 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:23:18.456 INFO:tasks.workunit.client.0.smithi162.stderr:194682: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:23:18.456 INFO:tasks.workunit.client.0.smithi162.stderr:194682: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:23:18.456 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:23:18.456 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:23:18.456 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:23:18.456 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:23:18.456 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:23:18.456 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:23:18.456 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: 1 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: 2 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: 3' 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836533 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836533 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533' 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350978 2024-03-09T13:23:18.457 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350978 2024-03-09T13:23:18.458 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-347892350978' 2024-03-09T13:23:18.458 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:20.699 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:23:20.699 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350979 2024-03-09T13:23:20.699 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350979 2024-03-09T13:23:20.699 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-347892350978 2-347892350979' 2024-03-09T13:23:20.699 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:20.699 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:23:20.699 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=274877906956 2024-03-09T13:23:20.699 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 274877906956 2024-03-09T13:23:20.699 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836533 1-347892350978 2-347892350979 3-274877906956' 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836533 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836533 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836533 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: waiting osd.0 seq 21474836533 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836533 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:23:20.700 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836533 2024-03-09T13:23:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-347892350978 2024-03-09T13:23:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:23:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-347892350978 2024-03-09T13:23:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350978 2024-03-09T13:23:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 347892350978' 2024-03-09T13:23:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:194682: waiting osd.1 seq 347892350978 2024-03-09T13:23:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:23:20.701 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350979 -lt 347892350978 2024-03-09T13:23:20.703 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in ceph tell osd.2 flush_pg_stats 2024-03-09T13:23:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350978 2024-03-09T13:23:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350978 2024-03-09T13:23:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-347892350979 2-347892350978' 2024-03-09T13:23:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:23:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=274877906957 2024-03-09T13:23:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 274877906957 2024-03-09T13:23:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836532 1-347892350979 2-347892350978 3-274877906957' 2024-03-09T13:23:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836532 2024-03-09T13:23:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836532 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836532 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: waiting osd.0 seq 21474836532 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836531 -lt 21474836532 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836534 -lt 21474836532 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-347892350979 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:20.705 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:23:20.706 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-347892350979 2024-03-09T13:23:20.706 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:20.706 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350979 2024-03-09T13:23:20.706 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 347892350979' 2024-03-09T13:23:20.706 INFO:tasks.workunit.client.0.smithi162.stderr:194684: waiting osd.1 seq 347892350979 2024-03-09T13:23:20.706 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:23:20.706 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350979 -lt 347892350979 2024-03-09T13:23:21.964 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seq$seqs 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350979 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350979 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350979 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350979' 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: waiting osd.2 seq 347892350979 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350979 -lt 347892350979 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-274877906956 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:21.965 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:23:21.966 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-274877906956 2024-03-09T13:23:21.966 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:21.966 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=274877906956 2024-03-09T13:23:21.966 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 274877906956' 2024-03-09T13:23:21.966 INFO:tasks.workunit.client.0.smithi162.stderr:194682: waiting osd.3 seq 274877906956 2024-03-09T13:23:21.966 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:23:21.966 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 274877906957 -lt 274877906956 2024-03-09T13:23:21.966 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:23:21.966 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:21.966 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:21.966 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:23:21.966 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:21.967 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:21.967 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:21.967 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:21.967 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:21.967 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:21.967 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:21.967 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:23:21.967 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:22.009 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtes 2024-03-09T13:23:22.009 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350978 2024-03-09T13:23:22.009 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:22.010 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:23:22.010 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350978 2024-03-09T13:23:22.010 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:22.010 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350978 2024-03-09T13:23:22.010 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350978' 2024-03-09T13:23:22.010 INFO:tasks.workunit.client.0.smithi162.stderr:194684: waiting osd.2 seq 347892350978 2024-03-09T13:23:22.010 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:23:22.010 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350979 -lt 347892350978 2024-03-09T13:23:22.010 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:22.010 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-274877906957 2024-03-09T13:23:22.010 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:22.010 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:23:22.010 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-274877906957 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=274877906957 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 274877906957' 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: waiting osd.3 seq 274877906957 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 274877906957 -lt 274877906957 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:22.011 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:22.012 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:22.012 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:22.012 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:22.012 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:23:22.012 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:23.463 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/cst/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:23.463 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:23.463 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:23:23.463 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T13:23:23.464 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:23:23.464 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T13:23:23.464 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:23:23.464 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:23:23.464 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:23:23.464 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:23.464 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:23.464 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:23.464 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:23.464 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:23.464 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:23.464 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:23.464 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:23:23.465 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:23:23.570 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntulone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:23.570 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:23.570 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:23:23.570 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T13:23:23.570 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:23:23.570 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T13:23:23.570 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:23:23.570 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:23:23.570 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:23:23.570 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:23:23.571 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:23:23.572 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:23:23.572 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:23:23.572 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:23:23.572 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:23:23.572 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:23:23.572 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:23:23.572 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:23:24.315 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cep/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:23:24.315 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:23:24.315 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:23:24.315 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:23:24.315 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:194682: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:194682: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:23:24.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:23:24.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 194681 2024-03-09T13:23:24.404 INFO:tasks.workunit.client.0.smithi162.stderr:htest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:23:24.404 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:23:24.404 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:23:24.404 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:23:24.404 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:24.404 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:24.405 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:24.405 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:24.405 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:24.405 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:24.405 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:24.405 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:23:24.405 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:24.405 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:24.405 INFO:tasks.workunit.client.0.smithi162.stderr:194684: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:24.405 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:23:24.405 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:23:24.405 INFO:tasks.workunit.client.0.smithi162.stderr:194684: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:23:24.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:23:24.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:23:24.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:23:24.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:23:24.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T13:23:24.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:23:24.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:23:24.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:23:24.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2024-03-09T13:23:24.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T13:23:24.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T13:23:24.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-09T13:23:24.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=1 2024-03-09T13:23:24.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T13:23:24.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:23:24.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:23:24.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:23:24.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 197602"' 2024-03-09T13:23:24.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 197602' 2024-03-09T13:23:24.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:23:24.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:23:24.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:23:24.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 197604"' 2024-03-09T13:23:24.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 197604' 2024-03-09T13:23:24.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:23:24.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T13:23:24.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 197602 197604' 2024-03-09T13:23:24.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:23:24.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:23:24.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 197602 2024-03-09T13:23:24.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/197605: /' 2024-03-09T13:23:24.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:23:24.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/197607: /' 2024-03-09T13:23:27.211 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:23:27.211 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:23:27.211 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:23:27.211 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:23:27.212 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:23:27.212 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:23:27.212 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:23:27.212 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:23:27.212 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:23:27.212 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:23:27.212 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:23:27.212 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:23:27.213 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:23:27.213 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:23:27.213 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:23:27.213 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:23:27.213 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:23:27.213 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:23:27.213 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:23:27.213 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:23:27.213 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:23:27.214 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:23:27.214 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T13:23:27.214 INFO:tasks.workunit.client.0.smithi162.stderr:197607: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:23:27.214 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:23:27.214 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:23:27.214 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:23:27.214 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:23:27.214 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:23:27.215 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:23:27.215 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:23:27.215 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:23:27.215 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:23:27.215 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:23:27.215 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:23:27.216 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:23:27.216 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:23:27.216 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:23:27.216 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:23:27.216 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:23:27.216 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:23:27.216 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:23:27.216 INFO:tasks.workunit.client.0.smithi162.stderr:197607: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:23:27.216 INFO:tasks.workunit.client.0.smithi162.stderr:197607: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:27.216 INFO:tasks.workunit.client.0.smithi162.stderr:197607: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:23:27.217 INFO:tasks.workunit.client.0.smithi162.stderr:197607: start osd.1 2024-03-09T13:23:27.218 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:23:27.483 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:23:27.484 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:23:27.485 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:23:27.486 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:23:27.486 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:23:27.486 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:27.487 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:23:27.488 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:23:27.488 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:23:27.488 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:23:27.488 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:23:27.488 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:23:27.488 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:23:27.488 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:23:27.488 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:23:27.488 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:23:27.488 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:23:27.488 INFO:tasks.workunit.client.0.smithi162.stderr:197605: start osd.3 2024-03-09T13:23:27.488 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:23:32.022 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: 2024-03-09T13:23:27.244+0000 7fcc5ae1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: 2024-03-09T13:23:27.246+0000 7fcc5ae1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: 2024-03-09T13:23:27.247+0000 7fcc5ae1a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:32.023 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: 0 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: 2024-03-09T13:23:27.768+0000 7fcc5ae1a640 -1 Falling back to public interface 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: 1 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:32.024 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:23:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:197607: 2 2024-03-09T13:23:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:23:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:197607: 2024-03-09T13:23:30.344+0000 7fcc5ae1a640 -1 osd.1 83 log_to_monitors true 2024-03-09T13:23:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:23:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:197607: 3 2024-03-09T13:23:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:23:32.025 INFO:tasks.workunit.client.0.smithi162.stderr:197607: osd.1 up in weight 1 up_from 88 up_thru 88 down_at 84 last_197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:23:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:23:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:23:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:23:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:23:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:197605: 2024-03-09T13:23:27.514+0000 7fefbd738640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:197605: 2024-03-09T13:23:27.517+0000 7fefbd738640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:197605: 2024-03-09T13:23:27.518+0000 7fefbd738640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:23:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:23:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:23:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:23:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:23:32.026 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: 0 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: 2024-03-09T13:23:28.044+0000 7fefbd738640 -1 Falling back to public interface 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: 1 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:32.027 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:23:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:197605: 2 2024-03-09T13:23:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:23:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:197605: 2024-03-09T13:23:30.679+0000 7fefbd738640 -1 osd.3 83 log_to_monitors true 2024-03-09T13:23:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:23:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:197605: 3 2024-03-09T13:23:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:32.028 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:23:32.895 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperclean_interval [81,83) [v2:127.0.0.1:6810/373270311,v1:127.0.0.1:6811/373270311] [v2:127.0.0.1:6812/373270311,v1:127.0.0.1:6813/373270311] exists,up 9dbfada3-cce6-4508-b697-011f9f8cf024 2024-03-09T13:23:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:23:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:23:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:23:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:23:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:23:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:23:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:23:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:23:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:197607: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:23:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:197607: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:23:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:23:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:23:32.896 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: 1 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: 2 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: 3' 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836537 2024-03-09T13:23:32.897 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836537 2024-03-09T13:23:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537' 2024-03-09T13:23:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:23:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-03-09T13:23:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-03-09T13:23:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-377957122050' 2024-03-09T13:23:32.898 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:33.773 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tells.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:33.773 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:33.773 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:33.773 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:23:33.773 INFO:tasks.workunit.client.0.smithi162.stderr:197605: 4 2024-03-09T13:23:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:23:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:197605: osd.3 up in weight 1 up_from 91 up_thru 91 down_at 84 last_clean_interval [64,83) [v2:127.0.0.1:6826/2638540171,v1:127.0.0.1:6827/2638540171] [v2:127.0.0.1:6828/2638540171,v1:127.0.0.1:6829/2638540171] exists,up 0fe1e115-9ef0-4727-a510-86a123c7da5e 2024-03-09T13:23:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:23:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:23:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:23:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:23:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:23:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:23:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:23:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:23:33.774 INFO:tasks.workunit.client.0.smithi162.stderr:197605: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: 1 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: 2 2024-03-09T13:23:33.775 INFO:tasks.workunit.client.0.smithi162.stderr:197605: 3' 2024-03-09T13:23:33.776 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:23:33.776 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:33.776 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:23:33.776 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836539 2024-03-09T13:23:33.776 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836539 2024-03-09T13:23:33.776 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539' 2024-03-09T13:23:33.776 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:34.637 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:23:34.637 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122051 2024-03-09T13:23:34.637 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122051 2024-03-09T13:23:34.637 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-377957122051' 2024-03-09T13:23:34.637 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:34.637 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:23:34.637 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350984 2024-03-09T13:23:34.637 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350984 2024-03-09T13:23:34.637 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-377957122051 2-347892350984' 2024-03-09T13:23:34.637 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:34.637 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:23:34.637 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023939 2024-03-09T13:23:34.638 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023939 2024-03-09T13:23:34.638 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836539 1-377957122051 2-347892350984 3-390842023939' 2024-03-09T13:23:34.638 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:34.638 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836539 2024-03-09T13:23:34.638 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:34.638 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:23:34.638 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836539 2024-03-09T13:23:34.638 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:34.638 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836539 2024-03-09T13:23:34.638 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-03-09T13:23:34.638 INFO:tasks.workunit.client.0.smithi162.stderr:197605: waiting osd.0 seq 21474836539 2024-03-09T13:23:34.638 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:34.638 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836539 2024-03-09T13:23:34.639 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:34.639 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122051 2024-03-09T13:23:34.639 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:34.639 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:23:34.639 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122051 2024-03-09T13:23:34.639 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:34.639 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122051 2024-03-09T13:23:34.639 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122051' 2024-03-09T13:23:34.639 INFO:tasks.workunit.client.0.smithi162.stderr:197605: waiting osd.1 seq 377957122051 2024-03-09T13:23:34.639 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:23:35.249 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221 osd.2 flush_pg_stats 2024-03-09T13:23:35.249 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350983 2024-03-09T13:23:35.249 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350983 2024-03-09T13:23:35.249 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-377957122050 2-347892350983' 2024-03-09T13:23:35.249 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:35.249 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:23:35.249 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023938 2024-03-09T13:23:35.250 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023938 2024-03-09T13:23:35.250 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836537 1-377957122050 2-347892350983 3-390842023938' 2024-03-09T13:23:35.250 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:35.250 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836537 2024-03-09T13:23:35.250 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:35.250 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:23:35.250 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836537 2024-03-09T13:23:35.250 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:35.250 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836537 2024-03-09T13:23:35.250 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-03-09T13:23:35.250 INFO:tasks.workunit.client.0.smithi162.stderr:197607: waiting osd.0 seq 21474836537 2024-03-09T13:23:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836536 -lt 21474836537 2024-03-09T13:23:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:23:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:23:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836539 -lt 21474836537 2024-03-09T13:23:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122050 2024-03-09T13:23:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:23:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122050 2024-03-09T13:23:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:35.252 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-03-09T13:23:35.252 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2024-03-09T13:23:35.252 INFO:tasks.workunit.client.0.smithi162.stderr:197607: waiting osd.1 seq 377957122050 2024-03-09T13:23:35.252 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:23:35.252 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122050 -lt 377957122050 2024-03-09T13:23:35.252 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:36.439 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350983 2024-03-09T13:23:36.439 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:36.439 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:23:36.439 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350983 2024-03-09T13:23:36.439 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:36.439 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350983 2024-03-09T13:23:36.440 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350983' 2024-03-09T13:23:36.440 INFO:tasks.workunit.client.0.smithi162.stderr:197607: waiting osd.2 seq 347892350983 2024-03-09T13:23:36.440 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:23:36.440 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350983 -lt 347892350983 2024-03-09T13:23:36.440 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:36.440 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-390842023938 2024-03-09T13:23:36.440 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:36.440 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:23:36.440 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-390842023938 2024-03-09T13:23:36.440 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:36.440 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023938 2024-03-09T13:23:36.440 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 390842023938' 2024-03-09T13:23:36.440 INFO:tasks.workunit.client.0.smithi162.stderr:197607: waiting osd.3 seq 390842023938 2024-03-09T13:23:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:23:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023938 -lt 390842023938 2024-03-09T13:23:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:23:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:23:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:36.441 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:36.442 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:23:36.442 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:36.778 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:36.778 INFO:tasks.workunit.client.0.smithi162.stderr:197607: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:36.778 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:23:36.778 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:23:36.778 INFO:tasks.workunit.client.0.smithi162.stderr:197607: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:23:36.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:23:38.154 INFO:tasks.workunit.client.0.smithi162.stderr:4: flush_pg_stats: test 377957122050 -lt 377957122051 2024-03-09T13:23:38.154 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:23:38.154 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:23:38.154 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:23:38.154 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122050 -lt 377957122051 2024-03-09T13:23:38.154 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:23:38.154 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:23:38.154 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:23:38.155 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122052 -lt 377957122051 2024-03-09T13:23:38.155 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:38.155 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350984 2024-03-09T13:23:38.155 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:38.155 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:23:38.155 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350984 2024-03-09T13:23:38.155 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:38.155 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350984 2024-03-09T13:23:38.155 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350984' 2024-03-09T13:23:38.155 INFO:tasks.workunit.client.0.smithi162.stderr:197605: waiting osd.2 seq 347892350984 2024-03-09T13:23:38.155 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:23:38.155 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350984 -lt 347892350984 2024-03-09T13:23:38.155 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-390842023939 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-390842023939 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023939 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 390842023939' 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: waiting osd.3 seq 390842023939 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023939 -lt 390842023939 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:23:38.156 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:38.768 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:38.768 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:38.768 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:38.768 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:38.768 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:38.768 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:38.768 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:23:38.768 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:38.768 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:38.768 INFO:tasks.workunit.client.0.smithi162.stderr:197605: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:38.768 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:23:38.769 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:23:38.769 INFO:tasks.workunit.client.0.smithi162.stderr:197605: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:23:38.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:23:38.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:23:38.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 197604 2024-03-09T13:23:38.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:23:38.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:23:38.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:23:38.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T13:23:38.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:23:38.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T13:23:38.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:23:38.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:23:38.770 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:23:38.770 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:23:39.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T13:23:39.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T13:23:39.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:23:39.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:23:39.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:23:39.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:23:39.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:23:39.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:23:39.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:23:04.465223+0000 2024-03-09T13:23:39.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:23:39.632 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:23:39.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:23:04.465223+0000 2024-03-09T13:23:39.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:23:39.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:23:04.465223+0000 2024-03-09T13:23:39.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:23:39.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:23:39.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:23:39.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:23:39.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:23:39.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:23:39.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:23:39.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:23:39.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:23:04.465223+0000 '>' 2024-03-09T13:23:04.465223+0000 2024-03-09T13:23:39.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:23:40.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:23:40.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:23:40.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:23:40.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:23:40.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:23:40.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:23:40.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:23:41.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:23:04.465223+0000 '>' 2024-03-09T13:23:04.465223+0000 2024-03-09T13:23:41.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:23:42.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:23:42.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:23:42.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:23:42.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:23:42.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:23:42.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:23:42.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:23:42.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:23:40.604434+0000 '>' 2024-03-09T13:23:04.465223+0000 2024-03-09T13:23:42.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 200451"' 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 200451' 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 200453"' 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 200453' 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T13:23:42.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 200451 200453' 2024-03-09T13:23:42.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:23:42.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:23:42.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 200451 2024-03-09T13:23:42.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/200454: /' 2024-03-09T13:23:42.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:23:42.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/200456: /' 2024-03-09T13:23:45.027 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:23:45.027 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:23:45.027 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:23:45.028 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: _ 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: hinfo_key 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: snapset 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:23:45.029 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:23:45.030 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:45.031 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:45.032 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:23:45.032 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:23:45.032 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:23:45.032 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:23:45.032 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:23:45.032 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:23:45.032 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:23:45.032 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:23:45.032 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:23:45.032 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:23:45.032 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:23:45.032 INFO:tasks.workunit.client.0.smithi162.stderr:200454: start osd.3 2024-03-09T13:23:45.635 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:23:45.636 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:23:45.636 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:23:45.636 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:23:45.636 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:23:45.636 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:23:45.636 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:23:45.636 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:23:45.636 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:23:45.636 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:23:45.636 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:23:45.637 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:23:45.637 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:23:45.637 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:23:45.637 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:23:45.637 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:23:45.637 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:23:45.637 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:23:45.637 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:23:45.637 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:23:45.637 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:23:45.637 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:23:45.638 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T13:23:45.638 INFO:tasks.workunit.client.0.smithi162.stderr:200456: _ 2024-03-09T13:23:45.638 INFO:tasks.workunit.client.0.smithi162.stderr:200456: hinfo_key 2024-03-09T13:23:45.638 INFO:tasks.workunit.client.0.smithi162.stderr:200456: snapset 2024-03-09T13:23:45.638 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:23:45.638 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:23:45.638 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:23:45.638 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:23:45.638 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:23:45.638 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:23:45.638 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:23:45.638 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:23:45.639 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:23:45.639 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:23:45.639 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:23:45.639 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:23:45.639 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:23:45.639 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:23:45.639 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:23:45.639 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:23:45.640 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:23:45.640 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:23:45.640 INFO:tasks.workunit.client.0.smithi162.stderr:200456: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:23:45.640 INFO:tasks.workunit.client.0.smithi162.stderr:200456: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:45.640 INFO:tasks.workunit.client.0.smithi162.stderr:200456: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:45.640 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:45.640 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:45.640 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:23:45.640 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:23:45.640 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:23:45.640 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:23:45.641 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:23:45.641 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:23:45.641 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:23:45.641 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:23:45.641 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:23:45.641 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:23:45.641 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:23:45.641 INFO:tasks.workunit.client.0.smithi162.stderr:200456: start osd.1 2024-03-09T13:23:49.195 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:23:49.195 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:23:49.195 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:23:49.196 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:23:49.196 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:23:49.196 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:23:49.196 INFO:tasks.workunit.client.0.smithi162.stderr:200454: 2024-03-09T13:23:45.059+0000 7f6b52224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:49.196 INFO:tasks.workunit.client.0.smithi162.stderr:200454: 2024-03-09T13:23:45.071+0000 7f6b52224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:49.196 INFO:tasks.workunit.client.0.smithi162.stderr:200454: 2024-03-09T13:23:45.072+0000 7f6b52224640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:49.196 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:23:49.196 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:23:49.196 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:23:49.196 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:23:49.197 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:23:49.197 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:49.197 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:23:49.197 INFO:tasks.workunit.client.0.smithi162.stderr:200454: 0 2024-03-09T13:23:49.197 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:49.197 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:23:49.197 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:49.197 INFO:tasks.workunit.client.0.smithi162.stderr:200454: 2024-03-09T13:23:45.597+0000 7f6b52224640 -1 Falling back to public interface 2024-03-09T13:23:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:23:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:200454: 1 2024-03-09T13:23:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: 2 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: 2024-03-09T13:23:48.145+0000 7f6b52224640 -1 osd.3 93 log_to_monitors true 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: 3 2024-03-09T13:23:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:50.228 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperovery_ops 2024-03-09T13:23:50.228 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:23:50.228 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:23:50.228 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:23:50.228 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:23:50.228 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:23:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:200456: 2024-03-09T13:23:45.663+0000 7f5ccafac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:200456: 2024-03-09T13:23:45.665+0000 7f5ccafac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:200456: 2024-03-09T13:23:45.667+0000 7f5ccafac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:23:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:23:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:23:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:23:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:23:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:23:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:200456: 0 2024-03-09T13:23:50.229 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: 2024-03-09T13:23:46.201+0000 7f5ccafac640 -1 Falling back to public interface 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: 1 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:23:50.230 INFO:tasks.workunit.client.0.smithi162.stderr:200456: 2 2024-03-09T13:23:50.231 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:50.231 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:23:50.231 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:23:50.231 INFO:tasks.workunit.client.0.smithi162.stderr:200456: 2024-03-09T13:23:48.849+0000 7f5ccafac640 -1 osd.1 93 log_to_monitors true 2024-03-09T13:23:50.231 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:23:50.231 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:23:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:23:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:200456: 3 2024-03-09T13:23:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:23:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:23:50.700 INFO:tasks.workunit.client.0.smithi162.stderr:200456: osd.1 up in weight 1 up_from 100 up_thru 88 down_s.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:23:50.700 INFO:tasks.workunit.client.0.smithi162.stderr:200454: osd.3 up in weight 1 up_from 98 up_thru 98 down_at 94 last_clean_interval [91,93) [v2:127.0.0.1:6810/580388721,v1:127.0.0.1:6811/580388721] [v2:127.0.0.1:6812/580388721,v1:127.0.0.1:6813/580388721] exists,up 0fe1e115-9ef0-4727-a510-86a123c7da5e 2024-03-09T13:23:50.700 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:23:50.700 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:23:50.700 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:23:50.700 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:23:50.700 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:23:50.700 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:23:50.700 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:23:50.700 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:23:50.700 INFO:tasks.workunit.client.0.smithi162.stderr:200454: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: 1 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: 2 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: 3' 2024-03-09T13:23:50.701 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:23:50.702 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:50.702 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:23:50.702 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836543 2024-03-09T13:23:50.702 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836543 2024-03-09T13:23:50.702 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543' 2024-03-09T13:23:50.702 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:50.702 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:23:50.702 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729602 2024-03-09T13:23:50.702 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729602 2024-03-09T13:23:50.702 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-429496729602' 2024-03-09T13:23:51.034 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $iat 94 last_clean_interval [88,93) [v2:127.0.0.1:6826/814290003,v1:127.0.0.1:6827/814290003] [v2:127.0.0.1:6828/814290003,v1:127.0.0.1:6829/814290003] exists,up 9dbfada3-cce6-4508-b697-011f9f8cf024 2024-03-09T13:23:51.034 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:23:51.035 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: 1 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: 2 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: 3' 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836544 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836544 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544' 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:51.036 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:23:51.037 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729603 2024-03-09T13:23:51.037 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729603 2024-03-09T13:23:51.037 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-429496729603' 2024-03-09T13:23:51.037 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:52.912 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats:ds 2024-03-09T13:23:52.912 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:23:52.912 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350988 2024-03-09T13:23:52.912 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350988 2024-03-09T13:23:52.912 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-429496729602 2-347892350988' 2024-03-09T13:23:52.912 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:52.912 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:23:52.912 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795010 2024-03-09T13:23:52.912 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795010 2024-03-09T13:23:52.913 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836543 1-429496729602 2-347892350988 3-420906795010' 2024-03-09T13:23:52.913 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:52.913 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836543 2024-03-09T13:23:52.913 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:52.913 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:23:52.913 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836543 2024-03-09T13:23:52.913 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:52.913 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836543 2024-03-09T13:23:52.913 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836543' 2024-03-09T13:23:52.913 INFO:tasks.workunit.client.0.smithi162.stderr:200454: waiting osd.0 seq 21474836543 2024-03-09T13:23:52.913 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:52.913 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836543 2024-03-09T13:23:52.913 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836543 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-429496729602 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-429496729602 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729602 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 429496729602' 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: waiting osd.1 seq 429496729602 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:23:52.914 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729603 -lt 429496729602 2024-03-09T13:23:53.282 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /h ceph tell osd.2 flush_pg_stats 2024-03-09T13:23:53.282 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=347892350989 2024-03-09T13:23:53.282 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 347892350989 2024-03-09T13:23:53.282 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-429496729603 2-347892350989' 2024-03-09T13:23:53.282 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:23:53.282 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:23:53.282 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795011 2024-03-09T13:23:53.282 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795011 2024-03-09T13:23:53.282 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836544 1-429496729603 2-347892350989 3-420906795011' 2024-03-09T13:23:53.282 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:53.283 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836544 2024-03-09T13:23:53.283 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:53.283 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:23:53.283 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836544 2024-03-09T13:23:53.283 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:53.283 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836544 2024-03-09T13:23:53.283 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-03-09T13:23:53.283 INFO:tasks.workunit.client.0.smithi162.stderr:200456: waiting osd.0 seq 21474836544 2024-03-09T13:23:53.283 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:53.283 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836542 -lt 21474836544 2024-03-09T13:23:53.283 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:23:53.283 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:23:53.283 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:23:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836544 -lt 21474836544 2024-03-09T13:23:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-429496729603 2024-03-09T13:23:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:23:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-429496729603 2024-03-09T13:23:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729603 2024-03-09T13:23:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 429496729603' 2024-03-09T13:23:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:200456: waiting osd.1 seq 429496729603 2024-03-09T13:23:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:23:53.284 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729603 -lt 429496729603 2024-03-09T13:23:54.216 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:54.216 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350988 2024-03-09T13:23:54.216 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:54.216 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:23:54.216 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350988 2024-03-09T13:23:54.216 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350988 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350988' 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: waiting osd.2 seq 347892350988 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350989 -lt 347892350988 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-420906795010 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-420906795010 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795010 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 420906795010' 2024-03-09T13:23:54.217 INFO:tasks.workunit.client.0.smithi162.stderr:200454: waiting osd.3 seq 420906795010 2024-03-09T13:23:54.218 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:23:54.218 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795011 -lt 420906795010 2024-03-09T13:23:54.218 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:23:54.218 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:54.218 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:54.218 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:23:54.218 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:54.218 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:54.218 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:54.218 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:54.218 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:54.218 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:54.218 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:54.219 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:23:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.clienqs 2024-03-09T13:23:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-347892350989 2024-03-09T13:23:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:23:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-347892350989 2024-03-09T13:23:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=347892350989 2024-03-09T13:23:54.522 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 347892350989' 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: waiting osd.2 seq 347892350989 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 347892350989 -lt 347892350989 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-420906795011 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-420906795011 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795011 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 420906795011' 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: waiting osd.3 seq 420906795011 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795011 -lt 420906795011 2024-03-09T13:23:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:23:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:23:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:23:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:55.695 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/t.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:55.695 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:55.695 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:55.695 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:23:55.695 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T13:23:55.695 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:23:55.695 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T13:23:55.695 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:23:55.695 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:23:55.697 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:23:55.697 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:23:55.697 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:23:55.697 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:23:55.697 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:23:55.697 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:23:55.697 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:23:55.697 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:23:55.697 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-09T13:23:55.996 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceclone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:55.996 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:55.996 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:23:55.996 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T13:23:55.997 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:23:55.997 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T13:23:55.997 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:23:55.997 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:23:55.997 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:23:55.997 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:55.997 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:55.997 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:55.997 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:55.997 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:55.997 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:55.997 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:23:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 5 2024-03-09T13:23:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:23:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:23:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:23:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:23:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:23:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:23:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:23:55.998 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:23:55.999 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1 2024-03-09T13:23:55.999 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-09T13:23:56.485 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtestph-helpers.sh:1340: get_is_making_recovery_progress: test 1 '!=' null 2024-03-09T13:23:56.485 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T13:23:56.485 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:23:56.486 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:23:56.486 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:23:56.486 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:56.486 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:56.486 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:56.486 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:56.486 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:56.486 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:56.486 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:56.486 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:23:56.486 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:56.486 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:56.486 INFO:tasks.workunit.client.0.smithi162.stderr:200454: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:56.487 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:23:56.487 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:23:56.487 INFO:tasks.workunit.client.0.smithi162.stderr:200454: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:23:56.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:23:56.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:23:56.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 200453 2024-03-09T13:23:56.747 INFO:tasks.workunit.client.0.smithi162.stderr:/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-03-09T13:23:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:23:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:23:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:23:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:23:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:23:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:23:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:23:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:23:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:23:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:23:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:23:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:23:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:23:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:200456: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:23:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:23:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:23:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:200456: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:23:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:23:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:23:56.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:23:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:23:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T13:23:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:23:56.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:23:56.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:23:56.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:23:56.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:23:56.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:23:56.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:23:56.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:23:56.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:23:56.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:23:56.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:23:56.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:23:56.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:23:56.899 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:23:56.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:23:56.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:23:56.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:23:56.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:23:56.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:23:56.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:23:56.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:23:56.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:23:56.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:23:56.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:23:56.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:23:56.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:23:56.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:23:56.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:56.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:56.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:23:56.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:23:56.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:23:56.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:23:56.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:23:56.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:23:56.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:23:56.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:23:56.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:23:56.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:23:56.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:23:56.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:23:56.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:23:56.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:23:56.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:23:56.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:23:56.937 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:23:56.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:23:56.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:23:56.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:23:56.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:23:56.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:23:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:23:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:23:56.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:23:56.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:23:56.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:23:56.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:23:56.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:23:56.944 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:23:56.944 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:56.944 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:56.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:23:56.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:23:56.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:23:56.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:23:56.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:23:56.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:56.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:56.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:23:56.950 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:23:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:23:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:23:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:23:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2024-03-09T13:23:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:842: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2024-03-09T13:23:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-03-09T13:23:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=false 2024-03-09T13:23:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-03-09T13:23:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-03-09T13:23:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:23:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:23:56.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:23:56.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:23:56.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:23:56.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T13:23:57.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:23:57.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:23:57.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:23:57.436 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:23:57.436 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:57.436 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:57.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:57.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T13:23:57.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:23:57.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:23:57.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:23:57.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:23:57.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:23:57.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:23:57.639 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:23:57.639 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:23:57.639 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:23:57.639 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:23:57.639 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:57.639 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:57.639 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:23:57.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:23:57.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:23:57.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:23:57.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:23:57.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:23:57.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:23:57.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:23:57.721 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:23:57.721 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:23:57.721 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:23:57.721 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:23:57.722 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:57.722 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:57.722 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:23:57.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:23:57.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:23:57.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-03-09T13:23:57.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:23:57.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:23:57.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:23:57.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:23:57.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:23:57.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:23:58.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:23:58.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:23:58.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:23:58.622 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:23:58.622 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:58.622 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:58.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:58.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:23:58.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:23:58.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-03-09T13:23:58.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:23:58.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-03-09T13:23:58.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:23:58.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:23:58.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:23:58.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:23:58.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:23:58.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:23:58.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:23:58.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:23:58.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:23:58.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:23:58.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:23:58.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:23:58.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:23:58.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:23:58.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:23:58.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:23:58.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:23:58.657 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:23:58.657 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:23:58.657 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:23:58.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:58.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:23:58.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:23:58.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:23:58.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:23:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:23:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:23:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:23:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:23:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:23:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:23:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:23:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:23:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:23:58.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:23:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5f32b9bf-26f6-4e02-bccd-dfe2df116ce9 2024-03-09T13:23:58.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5f32b9bf-26f6-4e02-bccd-dfe2df116ce9' 2024-03-09T13:23:58.659 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 5f32b9bf-26f6-4e02-bccd-dfe2df116ce9 2024-03-09T13:23:58.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:23:58.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDuYuxlBiIrKBAAVPRQdfQ0pQZrqaC/wBrv4g== 2024-03-09T13:23:58.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDuYuxlBiIrKBAAVPRQdfQ0pQZrqaC/wBrv4g=="}' 2024-03-09T13:23:58.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5f32b9bf-26f6-4e02-bccd-dfe2df116ce9 -i td/osd-scrub-repair/0/new.json 2024-03-09T13:23:58.877 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:23:58.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:23:58.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDuYuxlBiIrKBAAVPRQdfQ0pQZrqaC/wBrv4g== --osd-uuid 5f32b9bf-26f6-4e02-bccd-dfe2df116ce9 2024-03-09T13:23:58.916 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:23:58.917+0000 7f9397462640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:58.928 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:23:58.929+0000 7f9397462640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:58.929 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:23:58.930+0000 7f9397462640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:23:58.929 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:23:58.931+0000 7f9397462640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:23:58.930 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:23:58.931+0000 7f9397462640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:23:58.930 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:23:58.931+0000 7f9397462640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:23:59.029 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:23:59.030+0000 7f9397462640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:24:04.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:24:04.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:24:04.578 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:24:04.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:24:04.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:24:04.948 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:24:04.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:24:04.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:24:04.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:24:04.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:24:04.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:24:04.974 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:04.976+0000 7fb588e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:24:04.985 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:04.986+0000 7fb588e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:24:04.986 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:04.987+0000 7fb588e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:24:05.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:24:05.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:24:05.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:24:05.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:24:05.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:24:05.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:05.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:24:05.249 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:24:05.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:05.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:05.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:05.519 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:05.520+0000 7fb588e0d640 -1 Falling back to public interface 2024-03-09T13:24:06.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:06.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:06.520 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:24:06.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:24:06.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:06.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:06.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:07.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:07.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:07.807 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:24:07.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:24:07.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:07.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:07.845 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:07.846+0000 7fb588e0d640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:24:08.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:09.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:09.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:09.103 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:24:09.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:24:09.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:09.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:09.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:10.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:10.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:10.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:24:10.385 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:24:10.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:10.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:10.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:11.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:11.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:11.768 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:24:11.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:24:11.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:11.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:12.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:13.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:13.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:13.052 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:24:13.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:24:13.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:13.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:13.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:14.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:14.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:14.397 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:24:14.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:24:14.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:14.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:14.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:15.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:15.805 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:24:15.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:15.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:24:15.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:15.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:16.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:17.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:17.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:17.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:24:17.089 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:24:17.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:17.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:17.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:18.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:18.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:18.413 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:24:18.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:24:18.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:18.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:18.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:19.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:19.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:19.807 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:24:19.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:24:19.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:19.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:20.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:21.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:21.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:21.323 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:24:21.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:24:21.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:21.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:21.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:22.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:22.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:22.867 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:24:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:24:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:23.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:24.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:24.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:24.793 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:24:24.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:24:24.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:24.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:25.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:26.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:26.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:26.170 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:24:26.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:24:26.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:26.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:26.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:27.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:27.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:27.873 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:24:27.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:24:27.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:27.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:28.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:29.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:29.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:29.600 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:24:29.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:24:29.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:29.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:24:29.953 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4203310074,v1:127.0.0.1:6803/4203310074] [v2:127.0.0.1:6804/4203310074,v1:127.0.0.1:6805/4203310074] exists,up 5f32b9bf-26f6-4e02-bccd-dfe2df116ce9 2024-03-09T13:24:29.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:24:29.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:24:29.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:24:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:24:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-03-09T13:24:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:24:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:24:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:24:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:24:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:24:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:24:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:24:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:24:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:24:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:24:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:24:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:24:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:24:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:24:29.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:24:29.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:24:29.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:24:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:24:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:24:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:24:29.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:24:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:24:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:24:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:24:29.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:24:29.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:24:29.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:24:29.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:24:29.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:24:29.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:24:29.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:24:29.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:24:29.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:24:29.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:24:29.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:24:29.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=51280b07-0717-4ca8-a72e-0b03658a88b9 2024-03-09T13:24:29.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 51280b07-0717-4ca8-a72e-0b03658a88b9' 2024-03-09T13:24:29.961 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 51280b07-0717-4ca8-a72e-0b03658a88b9 2024-03-09T13:24:29.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:24:29.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANY+xlYE8hOhAA4cXJYJIVJHWMylM7hgCWcA== 2024-03-09T13:24:29.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANY+xlYE8hOhAA4cXJYJIVJHWMylM7hgCWcA=="}' 2024-03-09T13:24:29.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 51280b07-0717-4ca8-a72e-0b03658a88b9 -i td/osd-scrub-repair/1/new.json 2024-03-09T13:24:30.301 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:24:30.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:24:30.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANY+xlYE8hOhAA4cXJYJIVJHWMylM7hgCWcA== --osd-uuid 51280b07-0717-4ca8-a72e-0b03658a88b9 2024-03-09T13:24:30.341 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:30.341+0000 7fa801ec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:24:30.377 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:30.377+0000 7fa801ec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:24:30.378 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:30.378+0000 7fa801ec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:24:30.379 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:30.379+0000 7fa801ec6640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:24:30.379 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:30.379+0000 7fa801ec6640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:24:30.379 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:30.379+0000 7fa801ec6640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:24:30.452 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:30.452+0000 7fa801ec6640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:24:36.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:24:36.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:24:36.701 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:24:36.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:24:36.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:24:37.055 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:24:37.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:24:37.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:24:37.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:24:37.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:24:37.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:24:37.081 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:37.081+0000 7fcca1410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:24:37.083 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:37.084+0000 7fcca1410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:24:37.085 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:37.085+0000 7fcca1410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:24:37.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:24:37.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:24:37.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:24:37.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:24:37.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:24:37.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:37.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:24:37.331 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:24:37.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:37.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:37.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:37.611 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:37.611+0000 7fcca1410640 -1 Falling back to public interface 2024-03-09T13:24:38.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:38.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:38.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:24:38.603 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:24:38.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:38.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:38.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:39.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:39.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:39.890 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:24:39.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:24:39.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:39.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:40.042 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:24:40.042+0000 7fcca1410640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:24:40.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:41.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:41.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:41.244 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:24:41.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:24:41.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:41.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:41.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:42.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:42.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:42.527 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:24:42.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:24:42.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:42.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:42.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:43.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:43.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:43.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:24:43.921 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:24:43.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:43.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:44.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:45.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:45.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:45.209 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:24:45.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:24:45.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:45.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:45.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:46.495 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:24:46.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:24:46.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:46.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:46.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:47.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:47.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:47.887 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:24:47.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:24:47.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:47.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:48.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:49.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:49.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:49.173 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:24:49.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:24:49.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:49.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:49.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:50.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:50.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:50.457 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:24:50.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:24:50.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:50.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:50.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:51.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:51.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:51.879 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:24:51.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:24:51.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:51.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:52.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:53.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:53.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:53.556 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:24:53.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:24:53.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:53.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:54.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:55.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:55.292 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:24:55.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:55.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:24:55.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:55.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:55.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:56.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:56.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:56.995 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:24:56.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:24:56.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:56.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:57.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:24:58.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:24:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:24:58.604 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:24:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:24:58.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:24:58.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:24:59.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:00.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:00.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:00.208 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:25:00.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:25:00.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:00.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:25:00.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:01.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:01.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:01.911 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:25:01.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:25:01.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:01.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:25:02.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:03.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:03.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:03.204 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:25:03.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:25:03.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:03.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:25:03.478 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2404791335,v1:127.0.0.1:6811/2404791335] [v2:127.0.0.1:6812/2404791335,v1:127.0.0.1:6813/2404791335] exists,up 51280b07-0717-4ca8-a72e-0b03658a88b9 2024-03-09T13:25:03.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:25:03.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:25:03.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:25:03.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:25:03.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-03-09T13:25:03.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:25:03.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:25:03.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:25:03.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:25:03.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:25:03.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:25:03.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:25:03.485 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:25:03.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2abab101-a18a-4523-8af4-f512ce033cd9 2024-03-09T13:25:03.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2abab101-a18a-4523-8af4-f512ce033cd9' 2024-03-09T13:25:03.485 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 2abab101-a18a-4523-8af4-f512ce033cd9 2024-03-09T13:25:03.485 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:25:03.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAvY+xllkTQHRAASH9comDgzC3nlwgL9my9qg== 2024-03-09T13:25:03.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAvY+xllkTQHRAASH9comDgzC3nlwgL9my9qg=="}' 2024-03-09T13:25:03.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2abab101-a18a-4523-8af4-f512ce033cd9 -i td/osd-scrub-repair/2/new.json 2024-03-09T13:25:03.855 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:25:03.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T13:25:03.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvY+xllkTQHRAASH9comDgzC3nlwgL9my9qg== --osd-uuid 2abab101-a18a-4523-8af4-f512ce033cd9 2024-03-09T13:25:03.894 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:03.895+0000 7f0eb3e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:25:03.939 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:03.940+0000 7f0eb3e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:25:03.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:03.941+0000 7f0eb3e0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:25:03.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:03.942+0000 7f0eb3e0c640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:25:03.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:03.942+0000 7f0eb3e0c640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:25:03.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:03.942+0000 7f0eb3e0c640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:25:04.014 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:04.015+0000 7f0eb3e0c640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T13:25:10.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T13:25:10.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:25:10.024 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T13:25:10.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T13:25:10.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:25:10.374 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T13:25:10.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T13:25:10.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:25:10.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:25:10.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:25:10.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:25:10.401 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:10.401+0000 7fee0b29d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:25:10.403 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:10.403+0000 7fee0b29d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:25:10.404 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:10.404+0000 7fee0b29d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:25:10.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T13:25:10.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:25:10.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:25:10.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:25:10.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:25:10.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:10.651 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:25:10.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:25:10.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:10.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:10.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:10.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:10.941+0000 7fee0b29d640 -1 Falling back to public interface 2024-03-09T13:25:11.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:11.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:11.928 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:25:11.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:25:11.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:11.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:12.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:13.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:13.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:13.212 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:25:13.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:25:13.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:13.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:13.257 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:13.256+0000 7fee0b29d640 -1 osd.2 0 log_to_monitors true 2024-03-09T13:25:13.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:14.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:14.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:14.497 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:25:14.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:25:14.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:14.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:14.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:15.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:15.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:15.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:25:15.829 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:25:15.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:15.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:16.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:17.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:17.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:17.290 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:25:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:25:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:17.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:18.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:18.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:18.573 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:25:18.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:25:18.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:18.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:18.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:19.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:19.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:19.860 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:25:19.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:25:19.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:19.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:21.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:21.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:21.193 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:25:21.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:25:21.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:21.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:21.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:22.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:22.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:22.476 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:25:22.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:25:22.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:22.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:22.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:23.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:23.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:23.828 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:25:23.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:25:23.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:23.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:24.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:25.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:25.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:25.225 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:25:25.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:25:25.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:25.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:25.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:26.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:26.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:26.690 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:25:26.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:25:26.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:26.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:27.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:28.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:28.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:28.363 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:25:28.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:25:28.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:28.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:28.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:29.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:29.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:29.827 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:25:29.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:25:29.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:29.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:30.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:31.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:31.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:31.224 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:25:31.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:25:31.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:31.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:31.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:32.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:32.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:32.934 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:25:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:25:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:32.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:33.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:34.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:34.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:34.654 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:25:34.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:25:34.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:34.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:35.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:36.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:36.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:36.172 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:25:36.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:25:36.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:36.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:25:36.453 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2496096164,v1:127.0.0.1:6819/2496096164] [v2:127.0.0.1:6820/2496096164,v1:127.0.0.1:6821/2496096164] exists,up 2abab101-a18a-4523-8af4-f512ce033cd9 2024-03-09T13:25:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:25:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:25:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:25:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:25:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-03-09T13:25:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:25:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:25:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-09T13:25:36.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:25:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:25:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:25:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:25:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:25:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:25:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:25:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:25:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:25:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:25:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:25:36.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:25:36.459 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:25:36.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:25:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:25:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:25:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:25:36.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:25:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:25:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:25:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:25:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:25:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:25:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:25:36.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:25:36.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:25:36.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:25:36.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:25:36.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:25:36.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:25:36.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:25:36.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:25:36.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e89c8cb7-e250-4035-b64e-470d42b71834 2024-03-09T13:25:36.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 e89c8cb7-e250-4035-b64e-470d42b71834' 2024-03-09T13:25:36.462 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 e89c8cb7-e250-4035-b64e-470d42b71834 2024-03-09T13:25:36.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:25:36.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQY+xlM9RgHBAANrQrDxrIu+gqE8n0wiVHxw== 2024-03-09T13:25:36.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQY+xlM9RgHBAANrQrDxrIu+gqE8n0wiVHxw=="}' 2024-03-09T13:25:36.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e89c8cb7-e250-4035-b64e-470d42b71834 -i td/osd-scrub-repair/3/new.json 2024-03-09T13:25:36.875 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:25:36.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-09T13:25:36.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQY+xlM9RgHBAANrQrDxrIu+gqE8n0wiVHxw== --osd-uuid e89c8cb7-e250-4035-b64e-470d42b71834 2024-03-09T13:25:36.916 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:36.917+0000 7fe4b02b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:25:36.959 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:36.960+0000 7fe4b02b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:25:36.960 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:36.961+0000 7fe4b02b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:25:36.961 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:36.962+0000 7fe4b02b5640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T13:25:36.961 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:36.962+0000 7fe4b02b5640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T13:25:36.961 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:36.962+0000 7fe4b02b5640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T13:25:37.043 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:37.044+0000 7fe4b02b5640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-09T13:25:43.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-09T13:25:43.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:25:43.101 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-03-09T13:25:43.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-09T13:25:43.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:25:43.455 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-03-09T13:25:43.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-09T13:25:43.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:25:43.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:25:43.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:25:43.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:25:43.481 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:43.482+0000 7f5ed9a48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:25:43.483 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:43.484+0000 7f5ed9a48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:25:43.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:43.485+0000 7f5ed9a48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:25:43.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-09T13:25:43.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:25:43.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:25:43.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:25:43.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:25:43.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:43.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:25:43.729 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:25:43.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:43.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:25:44.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:44.018 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:44.019+0000 7f5ed9a48640 -1 Falling back to public interface 2024-03-09T13:25:45.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:45.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:45.004 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:25:45.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:25:45.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:45.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:25:45.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:46.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:46.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:46.285 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:25:46.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:25:46.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:46.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:25:46.380 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:25:46.381+0000 7f5ed9a48640 -1 osd.3 0 log_to_monitors true 2024-03-09T13:25:46.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:47.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:47.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:47.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:25:47.569 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:25:47.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:47.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:25:47.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:48.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:48.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:48.852 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:25:48.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:25:48.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:48.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:25:49.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:50.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:50.168 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:25:50.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:25:50.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:50.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:25:50.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:51.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:51.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:51.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:25:51.583 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:25:51.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:51.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:25:51.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:52.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:52.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:52.868 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:25:52.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:25:52.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:52.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:25:53.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:54.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:54.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:54.159 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:25:54.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:25:54.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:54.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:25:54.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:55.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:55.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:55.518 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:25:55.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:25:55.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:55.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:25:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:56.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:56.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:56.803 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:25:56.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:25:56.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:56.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:25:57.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:58.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:58.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:58.208 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:25:58.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:25:58.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:58.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:25:58.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:25:59.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:25:59.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:25:59.493 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:25:59.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:25:59.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:25:59.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:26:00.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:01.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:01.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:01.132 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:26:01.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:26:01.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:01.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:26:01.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:02.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:02.427 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:26:02.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:02.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:26:02.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:02.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:26:03.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:04.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:04.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:04.043 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:26:04.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:26:04.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:04.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:26:04.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:05.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:05.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:05.736 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:26:05.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:26:05.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:05.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:26:06.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:07.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:07.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:07.315 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:26:07.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:26:07.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:07.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:26:07.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:08.623 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:26:08.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:26:08.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:08.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:26:09.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:10.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:10.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:10.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:26:10.127 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:26:10.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:10.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:26:10.410 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1515115465,v1:127.0.0.1:6827/1515115465] [v2:127.0.0.1:6828/1515115465,v1:127.0.0.1:6829/1515115465] exists,up e89c8cb7-e250-4035-b64e-470d42b71834 2024-03-09T13:26:10.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:26:10.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:26:10.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:26:10.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:26:10.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-03-09T13:26:10.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:26:10.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:26:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-09T13:26:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:26:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-03-09T13:26:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:26:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:26:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:26:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:26:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-03-09T13:26:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-03-09T13:26:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:26:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:26:10.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:26:10.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:26:10.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:26:10.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:26:10.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:26:10.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:26:10.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:26:10.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-03-09T13:26:10.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:26:10.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=33df9bf5-b0d4-4ea0-9551-3ac6511fa78d 2024-03-09T13:26:10.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 33df9bf5-b0d4-4ea0-9551-3ac6511fa78d' 2024-03-09T13:26:10.417 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 33df9bf5-b0d4-4ea0-9551-3ac6511fa78d 2024-03-09T13:26:10.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:26:10.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQByY+xlxD/PGRAAgoUNJ0cL3BlAPn8AtKooEA== 2024-03-09T13:26:10.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQByY+xlxD/PGRAAgoUNJ0cL3BlAPn8AtKooEA=="}' 2024-03-09T13:26:10.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 33df9bf5-b0d4-4ea0-9551-3ac6511fa78d -i td/osd-scrub-repair/4/new.json 2024-03-09T13:26:10.828 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:26:10.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-03-09T13:26:10.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQByY+xlxD/PGRAAgoUNJ0cL3BlAPn8AtKooEA== --osd-uuid 33df9bf5-b0d4-4ea0-9551-3ac6511fa78d 2024-03-09T13:26:10.868 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:10.868+0000 7f568fe11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:26:10.938 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:10.937+0000 7f568fe11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:26:10.938 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:10.938+0000 7f568fe11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:26:10.939 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:10.939+0000 7f568fe11640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-09T13:26:10.939 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:10.939+0000 7f568fe11640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-09T13:26:10.939 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:10.939+0000 7f568fe11640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-09T13:26:11.012 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:11.012+0000 7f568fe11640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-03-09T13:26:16.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-03-09T13:26:16.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:26:16.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-09T13:26:16.972 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-03-09T13:26:16.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:26:17.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-09T13:26:17.329 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-03-09T13:26:17.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:26:17.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:26:17.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:26:17.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:26:17.355 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:17.355+0000 7f6b38f06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:26:17.357 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:17.357+0000 7f6b38f06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:26:17.358 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:17.358+0000 7f6b38f06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:26:17.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-09T13:26:17.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:26:17.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-09T13:26:17.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:26:17.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:26:17.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:17.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:26:17.606 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:26:17.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:17.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:17.879 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:17.879+0000 7f6b38f06640 -1 Falling back to public interface 2024-03-09T13:26:17.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:18.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:18.900 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:26:18.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:18.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:26:18.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:18.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:19.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:20.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:20.184 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:26:20.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:20.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:26:20.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:20.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:20.269 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:20.270+0000 7f6b38f06640 -1 osd.4 0 log_to_monitors true 2024-03-09T13:26:20.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:21.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:21.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:21.478 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:26:21.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:26:21.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:21.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:21.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:22.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:22.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:22.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:26:22.818 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:26:22.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:22.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:23.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:24.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:24.149 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:26:24.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:24.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:26:24.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:24.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:24.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:25.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:25.487 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:26:25.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:25.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:26:25.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:25.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:25.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:26.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:26.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:26.791 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:26:26.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:26:26.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:26.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:27.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:28.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:28.134 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:26:28.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:28.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:26:28.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:28.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:28.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:29.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:29.525 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:26:29.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:29.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:26:29.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:29.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:29.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:30.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:30.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:30.812 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:26:30.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:26:30.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:30.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:31.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:32.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:32.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:32.150 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:26:32.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:26:32.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:32.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:32.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:33.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:33.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:33.469 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:26:33.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:26:33.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:33.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:33.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:34.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:34.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:34.771 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:26:34.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:26:34.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:34.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:35.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:36.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:36.148 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:26:36.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:36.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:26:36.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:36.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:36.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:37.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:37.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:37.434 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:26:37.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:26:37.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:37.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:37.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:38.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:38.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:38.962 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:26:38.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:26:38.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:38.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:39.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:40.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:40.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:40.731 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:26:40.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:26:40.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:40.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:41.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:42.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:42.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:42.149 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:26:42.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:26:42.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:42.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:42.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:43.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:43.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:43.605 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:26:43.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:26:43.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:43.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:26:43.895 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/277846721,v1:127.0.0.1:6835/277846721] [v2:127.0.0.1:6836/277846721,v1:127.0.0.1:6837/277846721] exists,up 33df9bf5-b0d4-4ea0-9551-3ac6511fa78d 2024-03-09T13:26:43.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:26:43.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:26:43.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:26:43.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:26:43.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-03-09T13:26:43.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:26:43.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:26:43.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-09T13:26:43.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:26:43.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:26:43.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:26:43.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:26:43.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:26:43.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:26:43.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T13:26:43.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T13:26:43.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:26:43.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:26:43.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:26:43.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:26:43.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:26:43.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:26:43.900 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:26:43.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:26:43.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:26:43.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T13:26:43.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:26:43.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9fc21c30-bd92-43ad-a6a2-73fee02bea12 2024-03-09T13:26:43.902 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 9fc21c30-bd92-43ad-a6a2-73fee02bea12 2024-03-09T13:26:43.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 9fc21c30-bd92-43ad-a6a2-73fee02bea12' 2024-03-09T13:26:43.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:26:43.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTY+xlduB0NhAAe99XfFBGHPDbG2oM22ATFA== 2024-03-09T13:26:43.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTY+xlduB0NhAAe99XfFBGHPDbG2oM22ATFA=="}' 2024-03-09T13:26:43.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9fc21c30-bd92-43ad-a6a2-73fee02bea12 -i td/osd-scrub-repair/5/new.json 2024-03-09T13:26:44.272 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:26:44.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-03-09T13:26:44.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTY+xlduB0NhAAe99XfFBGHPDbG2oM22ATFA== --osd-uuid 9fc21c30-bd92-43ad-a6a2-73fee02bea12 2024-03-09T13:26:44.312 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:44.311+0000 7fc362ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:26:44.357 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:44.357+0000 7fc362ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:26:44.358 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:44.358+0000 7fc362ac8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:26:44.358 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:44.358+0000 7fc362ac8640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-09T13:26:44.358 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:44.358+0000 7fc362ac8640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-09T13:26:44.358 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:44.358+0000 7fc362ac8640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-09T13:26:44.424 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:44.423+0000 7fc362ac8640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-03-09T13:26:50.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-03-09T13:26:50.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:26:50.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-09T13:26:50.073 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-03-09T13:26:50.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:26:50.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-09T13:26:50.426 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-03-09T13:26:50.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:26:50.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:26:50.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:26:50.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:26:50.452 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:50.452+0000 7f66bba0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:26:50.454 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:50.454+0000 7f66bba0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:26:50.455 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:50.455+0000 7f66bba0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:26:50.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-09T13:26:50.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:26:50.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T13:26:50.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:26:50.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:26:50.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:50.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:26:50.707 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:26:50.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:50.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:26:50.974 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:50.974+0000 7f66bba0f640 -1 Falling back to public interface 2024-03-09T13:26:50.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:51.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:51.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:51.981 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:26:51.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:26:51.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:51.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:26:52.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:53.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:53.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:53.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:26:53.264 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:26:53.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:53.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:26:53.271 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:26:53.271+0000 7f66bba0f640 -1 osd.5 0 log_to_monitors true 2024-03-09T13:26:53.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:54.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:54.598 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:26:54.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:54.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:26:54.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:54.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:26:54.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:55.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:55.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:55.883 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:26:55.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:26:55.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:55.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:26:56.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:57.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:57.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:57.197 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:26:57.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:26:57.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:57.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:26:57.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:58.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:58.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:58.483 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:26:58.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:26:58.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:58.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:26:58.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:26:59.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:26:59.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:26:59.815 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:26:59.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:26:59.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:26:59.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:27:00.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:01.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:01.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:01.295 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:27:01.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:27:01.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:01.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:27:01.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:02.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:02.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:02.580 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:27:02.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:27:02.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:02.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:27:02.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:03.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:03.866 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:27:03.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:03.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:27:03.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:03.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:27:04.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:05.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:05.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:05.214 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:27:05.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:27:05.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:05.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:27:05.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:06.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:06.949 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:27:06.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:06.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:27:06.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:06.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:27:07.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:08.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:08.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:08.607 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:27:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:27:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:27:09.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:10.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:10.085 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:27:10.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:10.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:27:10.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:10.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:27:10.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:11.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:11.626 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:27:11.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:11.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:27:11.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:11.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:27:12.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:13.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:13.250 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:27:13.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:13.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:27:13.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:13.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:27:13.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:14.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:14.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:14.600 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:27:14.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:27:14.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:14.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:27:14.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:15.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:15.936 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:27:15.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:15.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:27:15.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:15.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:27:16.219 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3912368374,v1:127.0.0.1:6843/3912368374] [v2:127.0.0.1:6844/3912368374,v1:127.0.0.1:6845/3912368374] exists,up 9fc21c30-bd92-43ad-a6a2-73fee02bea12 2024-03-09T13:27:16.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:27:16.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:27:16.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:27:16.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:27:16.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-03-09T13:27:16.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:27:16.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:27:16.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-09T13:27:16.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:27:16.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-03-09T13:27:16.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:27:16.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:27:16.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:27:16.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:27:16.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-03-09T13:27:16.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:27:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:27:16.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:27:16.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:27:16.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:27:16.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:27:16.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:27:16.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:27:16.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:27:16.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:27:16.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-03-09T13:27:16.226 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:27:16.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8e26cbd9-d0ec-4da3-9291-f1d394a39731 2024-03-09T13:27:16.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 8e26cbd9-d0ec-4da3-9291-f1d394a39731' 2024-03-09T13:27:16.227 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 8e26cbd9-d0ec-4da3-9291-f1d394a39731 2024-03-09T13:27:16.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:27:16.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0Y+xlRfNaDhAAIgLKGZhrHlYttvkqLpN5gw== 2024-03-09T13:27:16.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0Y+xlRfNaDhAAIgLKGZhrHlYttvkqLpN5gw=="}' 2024-03-09T13:27:16.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8e26cbd9-d0ec-4da3-9291-f1d394a39731 -i td/osd-scrub-repair/6/new.json 2024-03-09T13:27:16.597 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:27:16.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-03-09T13:27:16.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0Y+xlRfNaDhAAIgLKGZhrHlYttvkqLpN5gw== --osd-uuid 8e26cbd9-d0ec-4da3-9291-f1d394a39731 2024-03-09T13:27:16.637 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:16.638+0000 7f5fd1ca6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:27:16.682 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:16.683+0000 7f5fd1ca6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:27:16.683 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:16.684+0000 7f5fd1ca6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:27:16.683 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:16.684+0000 7f5fd1ca6640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-09T13:27:16.684 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:16.684+0000 7f5fd1ca6640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-09T13:27:16.684 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:16.684+0000 7f5fd1ca6640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-09T13:27:16.774 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:16.774+0000 7f5fd1ca6640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-03-09T13:27:22.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-03-09T13:27:22.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:27:22.773 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-03-09T13:27:22.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-09T13:27:22.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:27:23.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-09T13:27:23.129 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-03-09T13:27:23.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:27:23.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:27:23.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:27:23.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:27:23.155 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:23.156+0000 7f843ae1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:27:23.157 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:23.158+0000 7f843ae1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:27:23.158 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:23.159+0000 7f843ae1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:27:23.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-09T13:27:23.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:27:23.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-09T13:27:23.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:27:23.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:27:23.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:23.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:27:23.406 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:27:23.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:23.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:23.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:23.676 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:23.677+0000 7f843ae1f640 -1 Falling back to public interface 2024-03-09T13:27:24.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:24.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:24.677 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:27:24.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:27:24.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:24.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:24.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:25.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:25.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:25.975 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:27:25.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:27:25.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:25.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:26.138 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:26.138+0000 7f843ae1f640 -1 osd.6 0 log_to_monitors true 2024-03-09T13:27:26.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:27.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:27.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:27.250 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:27:27.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:27:27.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:27.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:27.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:28.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:28.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:28.534 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:27:28.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:27:28.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:28.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:28.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:29.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:29.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:29.822 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:27:29.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:27:29.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:29.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:30.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:31.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:31.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:31.241 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:27:31.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:27:31.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:31.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:31.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:32.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:32.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:32.527 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:27:32.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:27:32.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:32.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:32.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:33.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:33.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:33.812 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:27:33.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:27:33.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:33.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:34.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:35.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:35.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:35.174 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:27:35.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:27:35.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:35.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:35.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:36.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:36.460 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:27:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:27:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:36.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:36.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:37.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:37.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:37.781 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:27:37.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:27:37.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:37.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:38.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:39.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:39.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:39.337 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:27:39.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:27:39.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:39.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:39.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:40.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:40.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:40.883 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:27:40.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:27:40.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:40.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:41.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:42.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:42.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:42.586 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:27:42.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:27:42.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:42.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:43.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:44.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:44.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:44.298 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:27:44.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:27:44.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:44.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:44.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:45.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:45.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:45.852 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:27:45.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:27:45.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:45.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:46.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:47.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:47.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:47.563 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:27:47.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:27:47.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:47.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:47.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:48.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:48.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:48.913 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:27:48.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:27:48.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:48.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:27:49.193 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1265858934,v1:127.0.0.1:6851/1265858934] [v2:127.0.0.1:6852/1265858934,v1:127.0.0.1:6853/1265858934] exists,up 8e26cbd9-d0ec-4da3-9291-f1d394a39731 2024-03-09T13:27:49.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:27:49.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:27:49.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:27:49.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:27:49.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-03-09T13:27:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:27:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:27:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-09T13:27:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:27:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-03-09T13:27:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:27:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:27:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:27:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:27:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-03-09T13:27:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-03-09T13:27:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:27:49.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:27:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:27:49.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:27:49.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:27:49.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:27:49.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:27:49.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:27:49.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:27:49.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-03-09T13:27:49.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:27:49.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4ca3a7cd-798c-46a5-8761-5f3cab9ea47e 2024-03-09T13:27:49.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 4ca3a7cd-798c-46a5-8761-5f3cab9ea47e' 2024-03-09T13:27:49.200 INFO:tasks.workunit.client.0.smithi162.stdout:add osd7 4ca3a7cd-798c-46a5-8761-5f3cab9ea47e 2024-03-09T13:27:49.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:27:49.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVY+xlFUboDBAAl6e8oLW28i1Pes2jrfTPqw== 2024-03-09T13:27:49.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVY+xlFUboDBAAl6e8oLW28i1Pes2jrfTPqw=="}' 2024-03-09T13:27:49.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4ca3a7cd-798c-46a5-8761-5f3cab9ea47e -i td/osd-scrub-repair/7/new.json 2024-03-09T13:27:49.618 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:27:49.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-03-09T13:27:49.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVY+xlFUboDBAAl6e8oLW28i1Pes2jrfTPqw== --osd-uuid 4ca3a7cd-798c-46a5-8761-5f3cab9ea47e 2024-03-09T13:27:49.658 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:49.658+0000 7ffa95d19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:27:49.728 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:49.728+0000 7ffa95d19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:27:49.729 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:49.729+0000 7ffa95d19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:27:49.729 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:49.729+0000 7ffa95d19640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-09T13:27:49.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:49.729+0000 7ffa95d19640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-09T13:27:49.730 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:49.729+0000 7ffa95d19640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-09T13:27:49.836 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:49.836+0000 7ffa95d19640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-03-09T13:27:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-03-09T13:27:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:27:55.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-09T13:27:55.792 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd7 key to auth repository 2024-03-09T13:27:55.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:27:56.145 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.7 2024-03-09T13:27:56.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-09T13:27:56.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:27:56.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:27:56.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:27:56.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:27:56.171 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:56.172+0000 7f067a013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:27:56.173 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:56.174+0000 7f067a013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:27:56.174 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:56.175+0000 7f067a013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:27:56.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-09T13:27:56.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:27:56.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-09T13:27:56.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:27:56.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:27:56.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:56.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:27:56.420 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:27:56.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:56.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:27:56.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:56.710 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:56.711+0000 7f067a013640 -1 Falling back to public interface 2024-03-09T13:27:57.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:57.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:57.693 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:27:57.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:27:57.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:57.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:27:57.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:27:58.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:27:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:27:58.976 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:27:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:27:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:27:58.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:27:59.107 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:27:59.107+0000 7f067a013640 -1 osd.7 0 log_to_monitors true 2024-03-09T13:27:59.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:00.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:00.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:00.316 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:28:00.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:28:00.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:00.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:00.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:01.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:01.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:01.599 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:28:01.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:28:01.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:01.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:01.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:02.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:02.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:02.882 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:28:02.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:28:02.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:02.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:03.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:04.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:04.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:04.308 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:28:04.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:28:04.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:04.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:04.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:05.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:05.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:05.595 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:28:05.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:28:05.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:05.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:06.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:07.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:07.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:07.009 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:28:07.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:28:07.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:07.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:07.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:08.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:08.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:08.296 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:28:08.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:28:08.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:08.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:08.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:09.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:09.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:09.580 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:28:09.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:28:09.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:09.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:09.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:10.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:10.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:10.866 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:28:10.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:28:10.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:10.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:11.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:12.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:12.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:12.209 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:28:12.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:28:12.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:12.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:12.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:13.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:13.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:13.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:28:13.685 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:28:13.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:13.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:14.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:15.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:15.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:15.181 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:28:15.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:28:15.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:15.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:15.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:16.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:16.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:16.590 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:28:16.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:28:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:16.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:17.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:18.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:18.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:18.026 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:28:18.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:28:18.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:18.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:18.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:19.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:19.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:19.464 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:28:19.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:28:19.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:19.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:19.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:20.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:20.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:20.857 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:28:20.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:28:20.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:20.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:21.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:22.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:22.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:22.302 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:28:22.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:28:22.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:22.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:28:22.590 INFO:tasks.workunit.client.0.smithi162.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1972288821,v1:127.0.0.1:6859/1972288821] [v2:127.0.0.1:6860/1972288821,v1:127.0.0.1:6861/1972288821] exists,up 4ca3a7cd-798c-46a5-8761-5f3cab9ea47e 2024-03-09T13:28:22.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:28:22.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:28:22.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:28:22.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:28:22.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-03-09T13:28:22.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:28:22.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:28:22.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-03-09T13:28:22.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:28:22.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-03-09T13:28:22.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:28:22.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:28:22.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:28:22.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:28:22.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-03-09T13:28:22.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-03-09T13:28:22.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:28:22.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:28:22.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:28:22.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:28:22.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:28:22.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:28:22.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:28:22.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:28:22.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:28:22.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-03-09T13:28:22.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:28:22.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=707a243e-93a4-4542-b792-58d3670abfd8 2024-03-09T13:28:22.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 707a243e-93a4-4542-b792-58d3670abfd8' 2024-03-09T13:28:22.597 INFO:tasks.workunit.client.0.smithi162.stdout:add osd8 707a243e-93a4-4542-b792-58d3670abfd8 2024-03-09T13:28:22.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:28:22.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD2Y+xl1fBwJBAASE8xecPBq6Op5bubd91CLQ== 2024-03-09T13:28:22.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD2Y+xl1fBwJBAASE8xecPBq6Op5bubd91CLQ=="}' 2024-03-09T13:28:22.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 707a243e-93a4-4542-b792-58d3670abfd8 -i td/osd-scrub-repair/8/new.json 2024-03-09T13:28:22.948 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:28:22.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-03-09T13:28:22.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2Y+xl1fBwJBAASE8xecPBq6Op5bubd91CLQ== --osd-uuid 707a243e-93a4-4542-b792-58d3670abfd8 2024-03-09T13:28:22.988 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:22.987+0000 7f6821d8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:28:22.999 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:22.998+0000 7f6821d8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:28:23.000 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:22.999+0000 7f6821d8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:28:23.000 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:23.000+0000 7f6821d8c640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-09T13:28:23.000 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:23.000+0000 7f6821d8c640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-09T13:28:23.000 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:23.000+0000 7f6821d8c640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-09T13:28:23.099 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:23.099+0000 7f6821d8c640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-03-09T13:28:29.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-03-09T13:28:29.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:28:29.179 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd8 key to auth repository 2024-03-09T13:28:29.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-03-09T13:28:29.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:28:29.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-03-09T13:28:29.537 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.8 2024-03-09T13:28:29.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:28:29.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:28:29.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:28:29.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:28:29.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:29.562+0000 7f03b9414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:28:29.564 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:29.564+0000 7f03b9414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:28:29.565 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:29.565+0000 7f03b9414640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:28:29.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-03-09T13:28:29.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:28:29.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-03-09T13:28:29.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:28:29.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:28:29.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:29.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:28:29.825 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:28:29.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:29.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:30.083 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:30.083+0000 7f03b9414640 -1 Falling back to public interface 2024-03-09T13:28:30.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:31.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:31.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:31.103 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:28:31.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:28:31.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:31.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:31.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:32.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:32.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:32.385 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:28:32.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:28:32.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:32.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:32.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:32.463+0000 7f03b9414640 -1 osd.8 0 log_to_monitors true 2024-03-09T13:28:32.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:33.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:33.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:33.666 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:28:33.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:28:33.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:33.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:34.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:35.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:35.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:35.110 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:28:35.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:28:35.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:35.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:35.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:36.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:36.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:36.438 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:28:36.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:28:36.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:36.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:36.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:37.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:37.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:37.724 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:28:37.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:28:37.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:37.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:38.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:39.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:39.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:39.018 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:28:39.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:28:39.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:39.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:39.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:40.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:40.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:40.420 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:28:40.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:28:40.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:40.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:40.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:41.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:41.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:41.704 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:28:41.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:28:41.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:41.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:42.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:43.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:43.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:43.025 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:28:43.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:28:43.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:43.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:43.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:44.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:44.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:44.428 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:28:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:28:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:44.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:44.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:45.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:45.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:45.714 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:28:45.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:28:45.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:45.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:46.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:47.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:47.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:47.238 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:28:47.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:28:47.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:47.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:47.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:48.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:48.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:48.605 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:28:48.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:28:48.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:48.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:49.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:50.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:50.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:50.093 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:28:50.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:28:50.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:50.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:50.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:51.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:51.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:51.683 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:28:51.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:28:51.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:51.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:52.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:53.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:53.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:53.217 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:28:53.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:28:53.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:53.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:53.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:54.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:54.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:54.921 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:28:54.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:28:54.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:54.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:55.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:28:56.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:28:56.240 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:28:56.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:28:56.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:28:56.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:28:56.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:28:56.522 INFO:tasks.workunit.client.0.smithi162.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/3995751367,v1:127.0.0.1:6867/3995751367] [v2:127.0.0.1:6868/3995751367,v1:127.0.0.1:6869/3995751367] exists,up 707a243e-93a4-4542-b792-58d3670abfd8 2024-03-09T13:28:56.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:28:56.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:28:56.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:28:56.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:28:56.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-03-09T13:28:56.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:28:56.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:28:56.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:28:56.527 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:28:56.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:28:56.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:28:56.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T13:28:56.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:28:56.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=822521c6-f894-4524-bb32-532c8efbf8d4 2024-03-09T13:28:56.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 822521c6-f894-4524-bb32-532c8efbf8d4' 2024-03-09T13:28:56.529 INFO:tasks.workunit.client.0.smithi162.stdout:add osd9 822521c6-f894-4524-bb32-532c8efbf8d4 2024-03-09T13:28:56.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:28:56.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAYZOxlOBtpIBAA60hKiu+kbVvIlCAzDmOxiw== 2024-03-09T13:28:56.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAYZOxlOBtpIBAA60hKiu+kbVvIlCAzDmOxiw=="}' 2024-03-09T13:28:56.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 822521c6-f894-4524-bb32-532c8efbf8d4 -i td/osd-scrub-repair/9/new.json 2024-03-09T13:28:56.912 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:28:56.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-03-09T13:28:56.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAYZOxlOBtpIBAA60hKiu+kbVvIlCAzDmOxiw== --osd-uuid 822521c6-f894-4524-bb32-532c8efbf8d4 2024-03-09T13:28:56.952 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:56.953+0000 7f70b2f06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:28:56.989 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:56.989+0000 7f70b2f06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:28:56.990 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:56.990+0000 7f70b2f06640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:28:56.991 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:56.991+0000 7f70b2f06640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-09T13:28:56.991 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:56.991+0000 7f70b2f06640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-09T13:28:56.991 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:56.991+0000 7f70b2f06640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-09T13:28:57.114 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:28:57.115+0000 7f70b2f06640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-03-09T13:29:03.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-03-09T13:29:03.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:29:03.349 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd9 key to auth repository 2024-03-09T13:29:03.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-03-09T13:29:03.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:29:03.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-03-09T13:29:03.708 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.9 2024-03-09T13:29:03.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:29:03.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:29:03.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:29:03.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:29:03.734 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:29:03.735+0000 7fdfebc99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:29:03.736 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:29:03.737+0000 7fdfebc99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:29:03.737 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:29:03.738+0000 7fdfebc99640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:29:03.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-03-09T13:29:03.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:29:03.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T13:29:03.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:29:03.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:29:03.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:03.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:29:03.988 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:29:03.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:03.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:04.253 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:29:04.254+0000 7fdfebc99640 -1 Falling back to public interface 2024-03-09T13:29:04.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:05.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:05.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:05.261 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:29:05.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:29:05.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:05.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:05.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:06.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:06.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:06.546 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:29:06.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:29:06.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:06.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:06.624 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:29:06.623+0000 7fdfebc99640 -1 osd.9 0 log_to_monitors true 2024-03-09T13:29:06.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:07.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:07.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:07.827 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:29:07.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:29:07.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:07.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:08.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:09.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:09.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:09.258 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:29:09.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:29:09.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:09.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:09.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:10.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:10.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:10.670 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:29:10.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:29:10.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:10.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:10.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:11.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:11.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:11.955 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:29:11.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:29:11.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:11.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:12.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:13.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:13.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:13.238 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:29:13.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:29:13.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:13.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:13.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:14.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:14.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:14.587 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:29:14.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:29:14.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:14.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:14.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:15.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:15.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:15.873 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:29:15.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:29:15.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:15.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:16.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:17.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:17.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:17.221 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:29:17.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:29:17.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:17.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:17.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:18.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:18.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:18.540 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:29:18.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:29:18.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:18.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:18.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:19.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:19.960 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:29:19.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:29:19.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:19.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:20.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:21.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:21.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:21.663 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:29:21.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:29:21.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:21.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:22.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:23.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:23.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:23.503 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:29:23.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:29:23.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:23.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:24.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:25.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:25.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:25.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:29:25.204 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:29:25.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:25.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:25.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:26.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:26.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:26.851 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:29:26.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:29:26.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:26.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:27.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:28.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:28.598 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:29:28.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:28.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:29:28.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:28.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:29.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:30.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:30.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:30.304 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:29:30.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:29:30.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:30.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:29:30.583 INFO:tasks.workunit.client.0.smithi162.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/1428408617,v1:127.0.0.1:6875/1428408617] [v2:127.0.0.1:6876/1428408617,v1:127.0.0.1:6877/1428408617] exists,up 822521c6-f894-4524-bb32-532c8efbf8d4 2024-03-09T13:29:30.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:29:30.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:29:30.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:29:30.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-03-09T13:29:30.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T13:29:30.850 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T13:29:30.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T13:29:30.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T13:29:31.364 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T13:29:31.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:29:32.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T13:29:35.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-03-09T13:29:35.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:29:35.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:29:35.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:29:35.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:29:35.650 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:29:35.650 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:29:35.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:29:35.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:29:35.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:29:35.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:29:35.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:29:35.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:29:35.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:29:35.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:29:35.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:29:36.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:29:36.086 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:29:36.086 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:29:36.086 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:29:36.086 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-03-09T13:29:36.086 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:29:36.086 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:29:36.086 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-03-09T13:29:36.086 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-03-09T13:29:36.086 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-03-09T13:29:36.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:29:36.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:36.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:29:36.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836548 2024-03-09T13:29:36.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836548 2024-03-09T13:29:36.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548' 2024-03-09T13:29:36.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:36.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:29:36.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673019 2024-03-09T13:29:36.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673019 2024-03-09T13:29:36.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673019' 2024-03-09T13:29:36.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:36.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:29:36.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509495 2024-03-09T13:29:36.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509495 2024-03-09T13:29:36.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673019 2-64424509495' 2024-03-09T13:29:36.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:36.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:29:36.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345967 2024-03-09T13:29:36.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345967 2024-03-09T13:29:36.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673019 2-64424509495 3-85899345967' 2024-03-09T13:29:36.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:36.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:29:36.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182437 2024-03-09T13:29:36.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182437 2024-03-09T13:29:36.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673019 2-64424509495 3-85899345967 4-107374182437' 2024-03-09T13:29:36.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:36.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:29:36.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018915 2024-03-09T13:29:36.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018915 2024-03-09T13:29:36.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673019 2-64424509495 3-85899345967 4-107374182437 5-128849018915' 2024-03-09T13:29:36.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:36.728 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:29:36.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855385 2024-03-09T13:29:36.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855385 2024-03-09T13:29:36.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673019 2-64424509495 3-85899345967 4-107374182437 5-128849018915 6-150323855385' 2024-03-09T13:29:36.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:36.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:29:36.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691858 2024-03-09T13:29:36.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691858 2024-03-09T13:29:36.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673019 2-64424509495 3-85899345967 4-107374182437 5-128849018915 6-150323855385 7-171798691858' 2024-03-09T13:29:36.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:36.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:29:37.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528332 2024-03-09T13:29:37.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528332 2024-03-09T13:29:37.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673019 2-64424509495 3-85899345967 4-107374182437 5-128849018915 6-150323855385 7-171798691858 8-193273528332' 2024-03-09T13:29:37.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:37.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:29:37.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364811 2024-03-09T13:29:37.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364811 2024-03-09T13:29:37.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836548 1-42949673019 2-64424509495 3-85899345967 4-107374182437 5-128849018915 6-150323855385 7-171798691858 8-193273528332 9-214748364811' 2024-03-09T13:29:37.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:37.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836548 2024-03-09T13:29:37.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:37.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:29:37.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836548 2024-03-09T13:29:37.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:37.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836548 2024-03-09T13:29:37.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2024-03-09T13:29:37.146 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836548 2024-03-09T13:29:37.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:29:37.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836548 -lt 21474836548 2024-03-09T13:29:37.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:37.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673019 2024-03-09T13:29:37.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:37.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:29:37.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673019 2024-03-09T13:29:37.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:37.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673019 2024-03-09T13:29:37.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673019' 2024-03-09T13:29:37.424 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673019 2024-03-09T13:29:37.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:29:37.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673019 -lt 42949673019 2024-03-09T13:29:37.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:37.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509495 2024-03-09T13:29:37.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:29:37.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509495 2024-03-09T13:29:37.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:37.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509495 2024-03-09T13:29:37.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509495' 2024-03-09T13:29:37.703 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509495 2024-03-09T13:29:37.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:29:37.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509495 -lt 64424509495 2024-03-09T13:29:37.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:37.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345967 2024-03-09T13:29:37.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:37.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:29:37.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345967 2024-03-09T13:29:37.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:37.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345967 2024-03-09T13:29:37.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345967' 2024-03-09T13:29:37.983 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345967 2024-03-09T13:29:37.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:29:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345967 -lt 85899345967 2024-03-09T13:29:38.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:38.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182437 2024-03-09T13:29:38.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:38.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:29:38.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182437 2024-03-09T13:29:38.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:38.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182437 2024-03-09T13:29:38.260 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182437 2024-03-09T13:29:38.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182437' 2024-03-09T13:29:38.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:29:38.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182437 -lt 107374182437 2024-03-09T13:29:38.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:38.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018915 2024-03-09T13:29:38.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:38.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:29:38.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018915 2024-03-09T13:29:38.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:38.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018915 2024-03-09T13:29:38.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018915' 2024-03-09T13:29:38.553 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 128849018915 2024-03-09T13:29:38.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:29:38.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018915 -lt 128849018915 2024-03-09T13:29:38.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:38.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855385 2024-03-09T13:29:38.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:38.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:29:38.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855385 2024-03-09T13:29:38.831 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:38.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855385 2024-03-09T13:29:38.832 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 150323855385 2024-03-09T13:29:38.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855385' 2024-03-09T13:29:38.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:29:39.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855385 -lt 150323855385 2024-03-09T13:29:39.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:39.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691858 2024-03-09T13:29:39.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:39.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:29:39.130 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691858 2024-03-09T13:29:39.130 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:39.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691858 2024-03-09T13:29:39.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691858' 2024-03-09T13:29:39.131 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691858 2024-03-09T13:29:39.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:29:39.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691859 -lt 171798691858 2024-03-09T13:29:39.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:39.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528332 2024-03-09T13:29:39.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:39.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:29:39.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528332 2024-03-09T13:29:39.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:39.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528332 2024-03-09T13:29:39.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528332' 2024-03-09T13:29:39.409 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528332 2024-03-09T13:29:39.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:29:39.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528332 -lt 193273528332 2024-03-09T13:29:39.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:39.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364811 2024-03-09T13:29:39.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:39.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:29:39.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364811 2024-03-09T13:29:39.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:39.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364811 2024-03-09T13:29:39.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364811' 2024-03-09T13:29:39.689 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 214748364811 2024-03-09T13:29:39.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:29:40.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364811 -lt 214748364811 2024-03-09T13:29:40.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:29:40.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:29:40.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:29:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T13:29:40.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:29:40.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:29:40.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:29:40.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:29:40.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:29:40.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:29:40.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:29:40.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:29:40.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:29:40.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:29:40.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:29:40.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T13:29:40.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:29:40.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:29:40.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2024-03-09T13:29:40.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T13:29:40.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T13:29:40.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-09T13:29:40.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T13:29:40.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-03-09T13:29:41.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T13:29:41.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T13:29:42.064 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' created 2024-03-09T13:29:42.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:29:43.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-09T13:29:43.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T13:29:43.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:29:43.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:29:43.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:29:43.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:29:43.078 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:29:43.078 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:29:43.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:29:43.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:29:43.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:29:43.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:29:43.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:29:43.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:29:43.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:29:43.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:29:43.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:29:43.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:29:43.453 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:29:43.453 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:29:43.454 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:29:43.454 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-03-09T13:29:43.454 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:29:43.454 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:29:43.454 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-03-09T13:29:43.454 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-03-09T13:29:43.454 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-03-09T13:29:43.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:29:43.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:43.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:29:43.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836550 2024-03-09T13:29:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836550 2024-03-09T13:29:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550' 2024-03-09T13:29:43.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:43.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:29:43.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673022 2024-03-09T13:29:43.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673022 2024-03-09T13:29:43.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022' 2024-03-09T13:29:43.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:43.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:29:43.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509498 2024-03-09T13:29:43.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509498 2024-03-09T13:29:43.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509498' 2024-03-09T13:29:43.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:43.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:29:44.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345969 2024-03-09T13:29:44.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345969 2024-03-09T13:29:44.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509498 3-85899345969' 2024-03-09T13:29:44.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:44.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:29:44.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182440 2024-03-09T13:29:44.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182440 2024-03-09T13:29:44.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509498 3-85899345969 4-107374182440' 2024-03-09T13:29:44.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:44.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:29:44.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018917 2024-03-09T13:29:44.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018917 2024-03-09T13:29:44.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509498 3-85899345969 4-107374182440 5-128849018917' 2024-03-09T13:29:44.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:44.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:29:44.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855388 2024-03-09T13:29:44.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855388 2024-03-09T13:29:44.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509498 3-85899345969 4-107374182440 5-128849018917 6-150323855388' 2024-03-09T13:29:44.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:44.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:29:44.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691861 2024-03-09T13:29:44.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691861 2024-03-09T13:29:44.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509498 3-85899345969 4-107374182440 5-128849018917 6-150323855388 7-171798691861' 2024-03-09T13:29:44.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:44.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:29:44.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528334 2024-03-09T13:29:44.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528334 2024-03-09T13:29:44.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509498 3-85899345969 4-107374182440 5-128849018917 6-150323855388 7-171798691861 8-193273528334' 2024-03-09T13:29:44.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:44.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:29:45.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364813 2024-03-09T13:29:45.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364813 2024-03-09T13:29:45.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836550 1-42949673022 2-64424509498 3-85899345969 4-107374182440 5-128849018917 6-150323855388 7-171798691861 8-193273528334 9-214748364813' 2024-03-09T13:29:45.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:45.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836550 2024-03-09T13:29:45.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:45.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:29:45.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836550 2024-03-09T13:29:45.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:45.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836550 2024-03-09T13:29:45.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2024-03-09T13:29:45.038 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836550 2024-03-09T13:29:45.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:29:45.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836550 -lt 21474836550 2024-03-09T13:29:45.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:45.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673022 2024-03-09T13:29:45.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:45.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:29:45.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673022 2024-03-09T13:29:45.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:45.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673022 2024-03-09T13:29:45.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673022' 2024-03-09T13:29:45.336 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673022 2024-03-09T13:29:45.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:29:45.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673022 -lt 42949673022 2024-03-09T13:29:45.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:45.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509498 2024-03-09T13:29:45.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:45.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:29:45.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509498 2024-03-09T13:29:45.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:45.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509498 2024-03-09T13:29:45.662 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509498 2024-03-09T13:29:45.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-03-09T13:29:45.663 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:29:46.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509498 -lt 64424509498 2024-03-09T13:29:46.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:46.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345969 2024-03-09T13:29:46.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:46.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:29:46.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345969 2024-03-09T13:29:46.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:46.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345969 2024-03-09T13:29:46.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345969' 2024-03-09T13:29:46.035 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345969 2024-03-09T13:29:46.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:29:46.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345969 -lt 85899345969 2024-03-09T13:29:46.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:46.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182440 2024-03-09T13:29:46.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:46.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:29:46.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182440 2024-03-09T13:29:46.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:46.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182440 2024-03-09T13:29:46.336 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182440 2024-03-09T13:29:46.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182440' 2024-03-09T13:29:46.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:29:46.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182440 -lt 107374182440 2024-03-09T13:29:46.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:46.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018917 2024-03-09T13:29:46.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:46.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:29:46.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018917 2024-03-09T13:29:46.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:46.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018917 2024-03-09T13:29:46.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018917' 2024-03-09T13:29:46.649 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 128849018917 2024-03-09T13:29:46.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:29:46.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018917 -lt 128849018917 2024-03-09T13:29:46.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:46.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855388 2024-03-09T13:29:46.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:46.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:29:46.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855388 2024-03-09T13:29:46.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:46.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855388 2024-03-09T13:29:46.973 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 150323855388 2024-03-09T13:29:46.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855388' 2024-03-09T13:29:46.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:29:47.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855388 -lt 150323855388 2024-03-09T13:29:47.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:47.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691861 2024-03-09T13:29:47.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:47.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:29:47.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691861 2024-03-09T13:29:47.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:47.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691861 2024-03-09T13:29:47.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691861' 2024-03-09T13:29:47.248 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691861 2024-03-09T13:29:47.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:29:47.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691861 -lt 171798691861 2024-03-09T13:29:47.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:47.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528334 2024-03-09T13:29:47.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:47.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:29:47.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528334 2024-03-09T13:29:47.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:47.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528334 2024-03-09T13:29:47.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528334' 2024-03-09T13:29:47.525 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528334 2024-03-09T13:29:47.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:29:47.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528335 -lt 193273528334 2024-03-09T13:29:47.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:47.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364813 2024-03-09T13:29:47.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:47.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:29:47.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364813 2024-03-09T13:29:47.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:47.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364813 2024-03-09T13:29:47.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364813' 2024-03-09T13:29:47.839 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 214748364813 2024-03-09T13:29:47.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:29:48.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364814 -lt 214748364813 2024-03-09T13:29:48.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:29:48.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:29:48.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:29:48.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:29:48.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:29:48.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:29:48.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:29:48.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:29:48.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:29:48.451 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:29:48.451 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:29:48.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:29:48.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:29:48.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:29:48.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:29:49.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:29:49.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:29:49.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:29:49.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T13:29:49.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-09T13:29:49.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T13:29:49.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-09T13:29:49.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-09T13:29:49.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:29:49.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:29:49.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-09T13:29:49.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:29:49.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:29:49.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:29:49.466 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:29:49.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:29:49.927 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:29:49.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:29:49.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:29:49.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T13:29:50.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-09T13:29:50.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T13:29:50.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:29:50.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:29:50.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:29:50.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-09T13:29:50.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-09T13:29:50.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T13:29:50.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-09T13:29:50.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-09T13:29:50.871 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:29:50.871 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T13:29:51.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-09T13:29:51.153 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:29:51.153 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-03-09T13:29:51.153 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:29:51.153 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:29:51.154 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:29:51.154 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:29:51.154 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-03-09T13:29:51.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-03-09T13:29:51.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-03-09T13:29:51.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-09T13:29:51.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-03-09T13:29:51.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-03-09T13:29:51.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-09T13:29:51.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T13:29:51.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T13:29:51.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-09T13:29:51.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:29:51.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:29:51.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:29:51.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:29:51.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:29:51.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:29:51.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:29:51.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:29:51.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:29:51.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:29:51.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:29:51.156 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:29:51.156 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:29:51.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:29:51.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:29:51.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:29:51.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:29:51.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:29:51.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:29:51.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:29:51.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:29:51.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:29:51.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:29:51.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T13:29:53.145 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T13:29:53.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:29:53.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:29:53.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:29:53.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:29:53.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:29:53.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:29:53.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:29:53.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:29:53.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:29:53.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:29:53.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:29:53.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:29:53.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:29:53.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:29:53.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:29:53.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:29:53.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:29:53.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:29:53.665 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:29:53.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:29:53.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:29:53.667 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-03-09T13:29:53.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:29:53.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:29:53.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:29:53.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:29:53.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:29:53.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:29:53.693 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:29:53.693+0000 7ff0e42b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:29:53.695 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:29:53.695+0000 7ff0e42b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:29:53.696 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:29:53.697+0000 7ff0e42b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:29:53.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:29:53.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:29:53.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:29:53.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:29:53.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:29:53.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:53.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:29:53.953 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:29:53.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:53.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:29:54.221 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:29:54.221+0000 7ff0e42b5640 -1 Falling back to public interface 2024-03-09T13:29:54.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:55.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:55.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:55.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:29:55.229 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:29:55.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:55.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:29:55.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:56.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:56.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:56.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:29:56.532 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:29:56.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:56.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:29:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:29:56.562+0000 7ff0e42b5640 -1 osd.3 69 log_to_monitors true 2024-03-09T13:29:56.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:29:57.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:29:57.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:29:57.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:29:57.822 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:29:57.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:29:57.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:29:58.169 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [20,69) [v2:127.0.0.1:6826/3268383617,v1:127.0.0.1:6827/3268383617] [v2:127.0.0.1:6828/3268383617,v1:127.0.0.1:6829/3268383617] exists,up e89c8cb7-e250-4035-b64e-470d42b71834 2024-03-09T13:29:58.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:29:58.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:29:58.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:29:58.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:29:58.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:29:58.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:29:58.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:29:58.169 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:29:58.169 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:29:58.169 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:29:58.169 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:29:58.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:29:58.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:29:58.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:29:58.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:29:58.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:29:58.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:29:58.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:29:58.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:29:58.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:29:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:29:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:29:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:29:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-03-09T13:29:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:29:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:29:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-03-09T13:29:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-03-09T13:29:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-03-09T13:29:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:29:58.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:58.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:29:58.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836554 2024-03-09T13:29:58.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836554 2024-03-09T13:29:58.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554' 2024-03-09T13:29:58.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:58.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:29:58.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673026 2024-03-09T13:29:58.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673026 2024-03-09T13:29:58.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673026' 2024-03-09T13:29:58.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:58.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:29:58.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509502 2024-03-09T13:29:58.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509502 2024-03-09T13:29:58.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673026 2-64424509502' 2024-03-09T13:29:58.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:58.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:29:59.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-03-09T13:29:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-03-09T13:29:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673026 2-64424509502 3-313532612610' 2024-03-09T13:29:59.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:59.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:29:59.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182444 2024-03-09T13:29:59.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182444 2024-03-09T13:29:59.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673026 2-64424509502 3-313532612610 4-107374182444' 2024-03-09T13:29:59.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:59.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:29:59.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018921 2024-03-09T13:29:59.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018921 2024-03-09T13:29:59.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673026 2-64424509502 3-313532612610 4-107374182444 5-128849018921' 2024-03-09T13:29:59.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:59.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:29:59.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855391 2024-03-09T13:29:59.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855391 2024-03-09T13:29:59.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673026 2-64424509502 3-313532612610 4-107374182444 5-128849018921 6-150323855391' 2024-03-09T13:29:59.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:59.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:29:59.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691865 2024-03-09T13:29:59.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691865 2024-03-09T13:29:59.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673026 2-64424509502 3-313532612610 4-107374182444 5-128849018921 6-150323855391 7-171798691865' 2024-03-09T13:29:59.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:59.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:29:59.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528338 2024-03-09T13:29:59.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528338 2024-03-09T13:29:59.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673026 2-64424509502 3-313532612610 4-107374182444 5-128849018921 6-150323855391 7-171798691865 8-193273528338' 2024-03-09T13:29:59.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:29:59.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:29:59.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364817 2024-03-09T13:29:59.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364817 2024-03-09T13:29:59.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836554 1-42949673026 2-64424509502 3-313532612610 4-107374182444 5-128849018921 6-150323855391 7-171798691865 8-193273528338 9-214748364817' 2024-03-09T13:29:59.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:59.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836554 2024-03-09T13:29:59.647 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:59.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:29:59.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836554 2024-03-09T13:29:59.649 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:59.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836554 2024-03-09T13:29:59.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2024-03-09T13:29:59.650 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836554 2024-03-09T13:29:59.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:29:59.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836554 -lt 21474836554 2024-03-09T13:29:59.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:29:59.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673026 2024-03-09T13:29:59.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:29:59.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:29:59.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673026 2024-03-09T13:29:59.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:29:59.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673026 2024-03-09T13:29:59.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673026' 2024-03-09T13:29:59.926 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673026 2024-03-09T13:29:59.926 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:30:00.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673025 -lt 42949673026 2024-03-09T13:30:00.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:30:01.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:30:01.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:30:01.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673026 -lt 42949673026 2024-03-09T13:30:01.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:01.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509502 2024-03-09T13:30:01.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:01.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:30:01.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509502 2024-03-09T13:30:01.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:01.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509502 2024-03-09T13:30:01.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509502' 2024-03-09T13:30:01.484 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509502 2024-03-09T13:30:01.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:30:01.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509502 -lt 64424509502 2024-03-09T13:30:01.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:01.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612610 2024-03-09T13:30:01.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:01.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:30:01.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612610 2024-03-09T13:30:01.761 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:01.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-03-09T13:30:01.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612610' 2024-03-09T13:30:01.762 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 313532612610 2024-03-09T13:30:01.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:30:02.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612610 2024-03-09T13:30:02.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:02.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182444 2024-03-09T13:30:02.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:02.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:30:02.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182444 2024-03-09T13:30:02.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:02.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182444 2024-03-09T13:30:02.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182444' 2024-03-09T13:30:02.052 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182444 2024-03-09T13:30:02.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:30:02.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182444 -lt 107374182444 2024-03-09T13:30:02.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:02.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018921 2024-03-09T13:30:02.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:02.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:30:02.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018921 2024-03-09T13:30:02.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:02.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018921 2024-03-09T13:30:02.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018921' 2024-03-09T13:30:02.328 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 128849018921 2024-03-09T13:30:02.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:30:02.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018921 -lt 128849018921 2024-03-09T13:30:02.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:02.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855391 2024-03-09T13:30:02.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:02.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:30:02.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855391 2024-03-09T13:30:02.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:02.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855391 2024-03-09T13:30:02.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855391' 2024-03-09T13:30:02.608 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 150323855391 2024-03-09T13:30:02.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:30:02.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855392 -lt 150323855391 2024-03-09T13:30:02.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:02.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691865 2024-03-09T13:30:02.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:02.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:30:02.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691865 2024-03-09T13:30:02.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:02.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691865 2024-03-09T13:30:02.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691865' 2024-03-09T13:30:02.889 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691865 2024-03-09T13:30:02.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:30:03.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691865 -lt 171798691865 2024-03-09T13:30:03.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:03.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528338 2024-03-09T13:30:03.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:03.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:30:03.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528338 2024-03-09T13:30:03.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:03.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528338 2024-03-09T13:30:03.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528338' 2024-03-09T13:30:03.179 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528338 2024-03-09T13:30:03.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:30:03.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528339 -lt 193273528338 2024-03-09T13:30:03.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:03.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364817 2024-03-09T13:30:03.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:03.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:30:03.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364817 2024-03-09T13:30:03.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364817 2024-03-09T13:30:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364817' 2024-03-09T13:30:03.455 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 214748364817 2024-03-09T13:30:03.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:30:03.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364818 -lt 214748364817 2024-03-09T13:30:03.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:30:03.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:30:03.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:30:04.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:30:04.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:30:04.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:30:04.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:30:04.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:30:04.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:30:04.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:30:04.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:30:04.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:30:04.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:30:04.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:30:04.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:30:04.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:30:04.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:30:04.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:30:04.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T13:30:04.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:30:04.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:30:04.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:30:04.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:30:05.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T13:30:05.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T13:30:05.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:30:05.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:30:05.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:30:05.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:30:05.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:30:05.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:30:05.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:29:41.854276+0000 2024-03-09T13:30:05.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:30:05.605 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:30:05.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:29:41.854276+0000 2024-03-09T13:30:05.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:30:05.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:29:41.854276+0000 2024-03-09T13:30:05.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:30:05.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:30:05.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:30:05.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:30:05.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:30:05.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:30:05.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:30:05.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:30:05.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:29:41.854276+0000 '>' 2024-03-09T13:29:41.854276+0000 2024-03-09T13:30:05.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:30:06.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:30:06.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:30:06.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:30:06.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:30:06.895 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:30:06.895 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:30:06.895 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:30:07.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:29:41.854276+0000 '>' 2024-03-09T13:29:41.854276+0000 2024-03-09T13:30:07.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:30:08.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:30:08.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:30:08.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:30:08.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:30:08.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:30:08.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:30:08.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:30:08.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:29:41.854276+0000 '>' 2024-03-09T13:29:41.854276+0000 2024-03-09T13:30:08.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:30:09.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:30:09.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:30:09.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:30:09.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:30:09.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:30:09.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:30:09.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:30:09.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:30:06.523069+0000 '>' 2024-03-09T13:29:41.854276+0000 2024-03-09T13:30:09.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:30:09.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:30:09.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:30:09.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:30:09.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T13:30:09.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:30:09.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:30:09.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:30:09.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:30:09.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T13:30:09.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:30:09.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T13:30:09.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:30:09.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:30:09.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:30:09.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:30:09.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:30:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:30:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:30:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:30:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:30:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T13:30:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:30:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:30:10.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-09T13:30:12.064 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:30:12.064 INFO:tasks.workunit.client.0.smithi162.stdout:hinfo_key 2024-03-09T13:30:12.064 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:30:12.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T13:30:12.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:30:12.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:30:12.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T13:30:12.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:30:12.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:30:12.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:30:12.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:30:12.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:30:12.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:30:12.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T13:30:12.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T13:30:12.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:30:12.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:30:12.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:30:12.342 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:30:12.342 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:30:12.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:30:12.343 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:30:12.343 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:30:12.343 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:30:12.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:30:12.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:30:12.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:30:12.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:30:12.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:30:12.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:30:12.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:30:12.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:30:12.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:30:12.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:30:12.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:30:12.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T13:30:12.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T13:30:12.344 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.9 2024-03-09T13:30:12.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:30:12.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T13:30:12.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T13:30:12.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:30:12.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:30:12.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:30:12.371 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:12.371+0000 7fde6320e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:30:12.373 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:12.373+0000 7fde6320e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:30:12.375 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:12.374+0000 7fde6320e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:30:12.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T13:30:12.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:30:12.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T13:30:12.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:30:12.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:30:12.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:30:12.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:30:12.631 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:30:12.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:30:12.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:30:12.902 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:12.901+0000 7fde6320e640 -1 Falling back to public interface 2024-03-09T13:30:12.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:30:13.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:30:13.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:30:13.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:30:13.955 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:30:13.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:30:13.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:30:14.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:30:15.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:30:15.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:30:15.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:30:15.240 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:30:15.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:30:15.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:30:15.250 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:15.249+0000 7fde6320e640 -1 osd.9 74 log_to_monitors true 2024-03-09T13:30:15.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:30:16.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:30:16.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:30:16.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:30:16.535 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:30:16.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:30:16.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:30:16.819 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 up in weight 1 up_from 78 up_thru 0 down_at 75 last_clean_interval [50,74) [v2:127.0.0.1:6874/1426623594,v1:127.0.0.1:6875/1426623594] [v2:127.0.0.1:6876/1426623594,v1:127.0.0.1:6877/1426623594] exists,up 822521c6-f894-4524-bb32-532c8efbf8d4 2024-03-09T13:30:16.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:30:16.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:30:16.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:30:16.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:30:16.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:30:16.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:30:16.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:30:16.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:30:16.820 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:30:16.820 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:30:16.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:30:16.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:30:16.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:30:16.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:30:16.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:30:16.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:30:16.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:30:16.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:30:16.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:30:17.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:30:17.261 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:30:17.261 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:30:17.261 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:30:17.261 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-03-09T13:30:17.261 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:30:17.261 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:30:17.261 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-03-09T13:30:17.261 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-03-09T13:30:17.261 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-03-09T13:30:17.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:30:17.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:17.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:30:17.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836559 2024-03-09T13:30:17.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836559 2024-03-09T13:30:17.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559' 2024-03-09T13:30:17.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:17.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:30:17.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673030 2024-03-09T13:30:17.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673030 2024-03-09T13:30:17.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030' 2024-03-09T13:30:17.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:17.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:30:17.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509507 2024-03-09T13:30:17.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509507 2024-03-09T13:30:17.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509507' 2024-03-09T13:30:17.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:17.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:30:17.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612615 2024-03-09T13:30:17.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612615 2024-03-09T13:30:17.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509507 3-313532612615' 2024-03-09T13:30:17.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:17.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:30:17.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182448 2024-03-09T13:30:17.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182448 2024-03-09T13:30:17.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509507 3-313532612615 4-107374182448' 2024-03-09T13:30:17.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:17.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:30:17.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018926 2024-03-09T13:30:17.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018926 2024-03-09T13:30:17.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509507 3-313532612615 4-107374182448 5-128849018926' 2024-03-09T13:30:17.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:17.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:30:18.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855396 2024-03-09T13:30:18.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855396 2024-03-09T13:30:18.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509507 3-313532612615 4-107374182448 5-128849018926 6-150323855396' 2024-03-09T13:30:18.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:18.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:30:18.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691870 2024-03-09T13:30:18.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691870 2024-03-09T13:30:18.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509507 3-313532612615 4-107374182448 5-128849018926 6-150323855396 7-171798691870' 2024-03-09T13:30:18.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:18.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:30:18.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528343 2024-03-09T13:30:18.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528343 2024-03-09T13:30:18.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509507 3-313532612615 4-107374182448 5-128849018926 6-150323855396 7-171798691870 8-193273528343' 2024-03-09T13:30:18.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:18.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:30:18.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449090 2024-03-09T13:30:18.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449090 2024-03-09T13:30:18.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836559 1-42949673030 2-64424509507 3-313532612615 4-107374182448 5-128849018926 6-150323855396 7-171798691870 8-193273528343 9-335007449090' 2024-03-09T13:30:18.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:18.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836559 2024-03-09T13:30:18.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:18.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:30:18.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836559 2024-03-09T13:30:18.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:18.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836559 2024-03-09T13:30:18.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836559' 2024-03-09T13:30:18.335 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836559 2024-03-09T13:30:18.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:30:18.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836558 -lt 21474836559 2024-03-09T13:30:18.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:30:19.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:30:19.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:30:19.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836559 -lt 21474836559 2024-03-09T13:30:19.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:19.895 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673030 2024-03-09T13:30:19.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:19.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:30:19.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673030 2024-03-09T13:30:19.897 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:19.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673030 2024-03-09T13:30:19.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673030' 2024-03-09T13:30:19.898 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673030 2024-03-09T13:30:19.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:30:20.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673031 -lt 42949673030 2024-03-09T13:30:20.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:20.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509507 2024-03-09T13:30:20.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:20.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:30:20.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509507 2024-03-09T13:30:20.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:20.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509507 2024-03-09T13:30:20.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509507' 2024-03-09T13:30:20.212 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509507 2024-03-09T13:30:20.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:30:20.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509507 -lt 64424509507 2024-03-09T13:30:20.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:20.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612615 2024-03-09T13:30:20.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:20.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:30:20.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612615 2024-03-09T13:30:20.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:20.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612615 2024-03-09T13:30:20.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612615' 2024-03-09T13:30:20.492 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 313532612615 2024-03-09T13:30:20.492 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:30:20.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612615 -lt 313532612615 2024-03-09T13:30:20.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:20.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182448 2024-03-09T13:30:20.767 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:20.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:30:20.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182448 2024-03-09T13:30:20.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:20.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182448 2024-03-09T13:30:20.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182448' 2024-03-09T13:30:20.769 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182448 2024-03-09T13:30:20.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:30:21.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182449 -lt 107374182448 2024-03-09T13:30:21.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:21.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018926 2024-03-09T13:30:21.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:21.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:30:21.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018926 2024-03-09T13:30:21.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:21.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018926 2024-03-09T13:30:21.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018926' 2024-03-09T13:30:21.057 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 128849018926 2024-03-09T13:30:21.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:30:21.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018926 -lt 128849018926 2024-03-09T13:30:21.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:21.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855396 2024-03-09T13:30:21.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:21.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:30:21.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855396 2024-03-09T13:30:21.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:21.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855396 2024-03-09T13:30:21.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855396' 2024-03-09T13:30:21.334 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 150323855396 2024-03-09T13:30:21.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:30:21.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855397 -lt 150323855396 2024-03-09T13:30:21.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:21.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691870 2024-03-09T13:30:21.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:21.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:30:21.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691870 2024-03-09T13:30:21.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:21.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691870 2024-03-09T13:30:21.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691870' 2024-03-09T13:30:21.612 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691870 2024-03-09T13:30:21.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:30:21.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691870 -lt 171798691870 2024-03-09T13:30:21.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:21.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528343 2024-03-09T13:30:21.906 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:21.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:30:21.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528343 2024-03-09T13:30:21.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:21.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528343 2024-03-09T13:30:21.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528343' 2024-03-09T13:30:21.909 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528343 2024-03-09T13:30:21.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:30:22.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528343 -lt 193273528343 2024-03-09T13:30:22.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:22.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-335007449090 2024-03-09T13:30:22.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:22.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:30:22.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-335007449090 2024-03-09T13:30:22.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:22.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449090 2024-03-09T13:30:22.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 335007449090' 2024-03-09T13:30:22.187 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 335007449090 2024-03-09T13:30:22.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:30:22.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449091 -lt 335007449090 2024-03-09T13:30:22.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:30:22.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:30:22.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:30:22.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:30:22.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:30:22.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:30:22.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:30:22.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:30:22.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:30:22.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:30:22.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:30:23.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:30:23.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:30:23.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:30:23.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:30:23.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:30:23.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:30:23.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:30:23.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:30:23.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:30:23.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-03-09T13:30:23.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T13:30:23.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:30:23.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:30:23.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:30:23.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:30:23.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:30:23.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:30:23.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:30:23.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:30:23.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T13:30:23.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:30:23.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:30:23.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-09T13:30:25.850 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:30:26.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T13:30:26.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:30:26.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:30:26.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T13:30:26.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:30:26.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:30:26.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:30:26.377 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T13:30:26.378 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.5 2024-03-09T13:30:26.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:30:26.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T13:30:26.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T13:30:26.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:30:26.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:30:26.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:30:26.405 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:26.406+0000 7f973780d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:30:26.408 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:26.408+0000 7f973780d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:30:26.409 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:26.410+0000 7f973780d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:30:26.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T13:30:26.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:30:26.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T13:30:26.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:30:26.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:30:26.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:30:26.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:30:26.666 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:30:26.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:30:26.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:30:26.925 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:26.926+0000 7f973780d640 -1 Falling back to public interface 2024-03-09T13:30:26.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:30:27.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:30:27.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:30:27.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:30:27.942 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:30:27.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:30:27.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:30:28.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:30:29.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:30:29.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:30:29.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:30:29.228 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:30:29.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:30:29.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:30:29.403 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:29.403+0000 7f973780d640 -1 osd.5 79 log_to_monitors true 2024-03-09T13:30:29.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:30:30.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:30:30.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:30:30.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:30:30.506 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:30:30.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:30:30.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:30:30.817 INFO:tasks.workunit.client.0.smithi162.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 80 last_clean_interval [30,79) [v2:127.0.0.1:6842/1891603657,v1:127.0.0.1:6843/1891603657] [v2:127.0.0.1:6844/1891603657,v1:127.0.0.1:6845/1891603657] exists,up 9fc21c30-bd92-43ad-a6a2-73fee02bea12 2024-03-09T13:30:30.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:30:30.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:30:30.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:30:30.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:30:30.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:30:30.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:30:30.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:30:30.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:30:30.818 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:30:30.818 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:30:30.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:30:30.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:30:30.819 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:30:30.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:30:30.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:30:30.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:30:30.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:30:30.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:30:30.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:30:31.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:30:31.305 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:30:31.305 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:30:31.305 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:30:31.305 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-03-09T13:30:31.305 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:30:31.305 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:30:31.306 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-03-09T13:30:31.306 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-03-09T13:30:31.306 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-03-09T13:30:31.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:30:31.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:31.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:30:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836563 2024-03-09T13:30:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836563 2024-03-09T13:30:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563' 2024-03-09T13:30:31.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:31.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:30:31.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673034 2024-03-09T13:30:31.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673034 2024-03-09T13:30:31.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673034' 2024-03-09T13:30:31.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:31.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:30:31.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509511 2024-03-09T13:30:31.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509511 2024-03-09T13:30:31.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673034 2-64424509511' 2024-03-09T13:30:31.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:31.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:30:31.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612619 2024-03-09T13:30:31.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612619 2024-03-09T13:30:31.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673034 2-64424509511 3-313532612619' 2024-03-09T13:30:31.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:31.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:30:32.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182452 2024-03-09T13:30:32.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182452 2024-03-09T13:30:32.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673034 2-64424509511 3-313532612619 4-107374182452' 2024-03-09T13:30:32.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:32.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:30:32.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-03-09T13:30:32.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-03-09T13:30:32.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673034 2-64424509511 3-313532612619 4-107374182452 5-360777252866' 2024-03-09T13:30:32.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:32.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:30:32.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855400 2024-03-09T13:30:32.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855400 2024-03-09T13:30:32.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673034 2-64424509511 3-313532612619 4-107374182452 5-360777252866 6-150323855400' 2024-03-09T13:30:32.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:32.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:30:32.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691873 2024-03-09T13:30:32.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691873 2024-03-09T13:30:32.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673034 2-64424509511 3-313532612619 4-107374182452 5-360777252866 6-150323855400 7-171798691873' 2024-03-09T13:30:32.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:32.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:30:32.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528347 2024-03-09T13:30:32.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528347 2024-03-09T13:30:32.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673034 2-64424509511 3-313532612619 4-107374182452 5-360777252866 6-150323855400 7-171798691873 8-193273528347' 2024-03-09T13:30:32.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:32.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:30:32.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449094 2024-03-09T13:30:32.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449094 2024-03-09T13:30:32.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836563 1-42949673034 2-64424509511 3-313532612619 4-107374182452 5-360777252866 6-150323855400 7-171798691873 8-193273528347 9-335007449094' 2024-03-09T13:30:32.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:32.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836563 2024-03-09T13:30:32.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:32.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:30:32.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836563 2024-03-09T13:30:32.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:32.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836563 2024-03-09T13:30:32.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836563' 2024-03-09T13:30:32.575 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836563 2024-03-09T13:30:32.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:30:32.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836562 -lt 21474836563 2024-03-09T13:30:32.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:30:33.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:30:33.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:30:34.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836563 -lt 21474836563 2024-03-09T13:30:34.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:34.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673034 2024-03-09T13:30:34.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:34.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:30:34.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673034 2024-03-09T13:30:34.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:34.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673034 2024-03-09T13:30:34.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673034' 2024-03-09T13:30:34.139 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673034 2024-03-09T13:30:34.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:30:34.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673035 -lt 42949673034 2024-03-09T13:30:34.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:34.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509511 2024-03-09T13:30:34.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:34.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:30:34.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509511 2024-03-09T13:30:34.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:34.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509511 2024-03-09T13:30:34.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509511' 2024-03-09T13:30:34.417 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509511 2024-03-09T13:30:34.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:30:34.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509511 -lt 64424509511 2024-03-09T13:30:34.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:34.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612619 2024-03-09T13:30:34.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:34.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:30:34.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612619 2024-03-09T13:30:34.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:34.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612619 2024-03-09T13:30:34.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612619' 2024-03-09T13:30:34.694 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 313532612619 2024-03-09T13:30:34.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:30:34.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612619 -lt 313532612619 2024-03-09T13:30:34.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:34.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182452 2024-03-09T13:30:34.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:34.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:30:34.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182452 2024-03-09T13:30:34.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:34.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182452 2024-03-09T13:30:34.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182452' 2024-03-09T13:30:34.973 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182452 2024-03-09T13:30:34.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:30:35.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182453 -lt 107374182452 2024-03-09T13:30:35.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:35.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252866 2024-03-09T13:30:35.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:35.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:30:35.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252866 2024-03-09T13:30:35.250 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-03-09T13:30:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-03-09T13:30:35.251 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 360777252866 2024-03-09T13:30:35.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:30:35.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252866 2024-03-09T13:30:35.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:35.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855400 2024-03-09T13:30:35.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:35.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:30:35.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855400 2024-03-09T13:30:35.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:35.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855400 2024-03-09T13:30:35.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855400' 2024-03-09T13:30:35.532 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 150323855400 2024-03-09T13:30:35.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:30:35.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855401 -lt 150323855400 2024-03-09T13:30:35.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:35.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691873 2024-03-09T13:30:35.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:35.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:30:35.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691873 2024-03-09T13:30:35.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:35.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691873 2024-03-09T13:30:35.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2024-03-09T13:30:35.838 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691873 2024-03-09T13:30:35.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:30:36.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691874 -lt 171798691873 2024-03-09T13:30:36.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:36.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528347 2024-03-09T13:30:36.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:36.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:30:36.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528347 2024-03-09T13:30:36.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:36.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528347 2024-03-09T13:30:36.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528347' 2024-03-09T13:30:36.114 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528347 2024-03-09T13:30:36.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:30:36.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528347 -lt 193273528347 2024-03-09T13:30:36.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:36.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-335007449094 2024-03-09T13:30:36.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:36.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:30:36.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-335007449094 2024-03-09T13:30:36.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:36.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449094 2024-03-09T13:30:36.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 335007449094' 2024-03-09T13:30:36.391 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 335007449094 2024-03-09T13:30:36.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:30:36.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449094 -lt 335007449094 2024-03-09T13:30:36.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:30:36.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:30:36.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:30:37.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:30:37.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:30:37.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:30:37.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:30:37.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:30:37.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:30:37.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:30:37.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:30:37.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:30:37.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:30:37.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:30:37.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:30:37.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:30:37.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:30:37.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:30:37.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T13:30:37.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:30:37.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:30:37.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:30:37.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:30:37.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T13:30:37.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T13:30:37.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:30:37.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:30:37.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:30:37.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:30:37.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:30:37.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:30:38.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:30:06.523069+0000 2024-03-09T13:30:38.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:30:38.487 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:30:38.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:30:06.523069+0000 2024-03-09T13:30:38.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:30:38.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:30:06.523069+0000 2024-03-09T13:30:38.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:30:38.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:30:38.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:30:38.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:30:38.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:30:38.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:30:38.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:30:38.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:30:38.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:30:06.523069+0000 '>' 2024-03-09T13:30:06.523069+0000 2024-03-09T13:30:38.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:30:39.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:30:39.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:30:39.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:30:39.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:30:39.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:30:39.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:30:39.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:30:40.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:30:06.523069+0000 '>' 2024-03-09T13:30:06.523069+0000 2024-03-09T13:30:40.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:30:41.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:30:41.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:30:41.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:30:41.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:30:41.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:30:41.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:30:41.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:30:41.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:30:06.523069+0000 '>' 2024-03-09T13:30:06.523069+0000 2024-03-09T13:30:41.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:30:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:30:42.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:30:42.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:30:42.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:30:42.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:30:42.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:30:42.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:30:42.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:30:06.523069+0000 '>' 2024-03-09T13:30:06.523069+0000 2024-03-09T13:30:42.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:30:43.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:30:43.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:30:43.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:30:43.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:30:43.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:30:43.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:30:43.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:30:43.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:30:38.546280+0000 '>' 2024-03-09T13:30:06.523069+0000 2024-03-09T13:30:43.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:30:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:30:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:30:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:30:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T13:30:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:30:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:30:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:30:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:30:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T13:30:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:30:43.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T13:30:43.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:30:43.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:30:43.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:30:43.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:30:43.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:30:44.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:30:44.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:30:44.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:30:44.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:30:44.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T13:30:44.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:30:44.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:30:44.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-09T13:30:46.358 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:30:46.359 INFO:tasks.workunit.client.0.smithi162.stdout:hinfo_key 2024-03-09T13:30:46.359 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:30:46.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T13:30:46.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:30:46.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:30:46.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T13:30:46.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:30:46.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:30:46.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:30:46.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:30:46.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:30:46.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:30:46.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:30:46.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:30:46.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:30:46.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:30:46.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:30:46.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:30:46.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:30:46.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:30:46.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:30:46.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T13:30:46.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T13:30:46.635 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.9 2024-03-09T13:30:46.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:30:46.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T13:30:46.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T13:30:46.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:30:46.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:30:46.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:30:46.663 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:46.662+0000 7fe13dc8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:30:46.665 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:46.665+0000 7fe13dc8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:30:46.666 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:46.666+0000 7fe13dc8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:30:46.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T13:30:46.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:30:46.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T13:30:46.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:30:46.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:30:46.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:30:46.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:30:46.949 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:30:46.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:30:46.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:30:47.185 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:47.185+0000 7fe13dc8b640 -1 Falling back to public interface 2024-03-09T13:30:47.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:30:48.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:30:48.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:30:48.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:30:48.224 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:30:48.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:30:48.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:30:48.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:30:49.462 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:30:49.462+0000 7fe13dc8b640 -1 osd.9 87 log_to_monitors true 2024-03-09T13:30:49.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:30:49.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:30:49.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:30:49.510 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:30:49.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:30:49.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:30:49.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:30:50.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:30:50.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:30:50.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:30:50.862 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:30:50.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:30:50.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:30:51.411 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 up in weight 1 up_from 91 up_thru 0 down_at 87 last_clean_interval [78,87) [v2:127.0.0.1:6874/712571271,v1:127.0.0.1:6875/712571271] [v2:127.0.0.1:6876/712571271,v1:127.0.0.1:6877/712571271] exists,up 822521c6-f894-4524-bb32-532c8efbf8d4 2024-03-09T13:30:51.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:30:51.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:30:51.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:30:51.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:30:51.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:30:51.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:30:51.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:30:51.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:30:51.412 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:30:51.412 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:30:51.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:30:51.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:30:51.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:30:51.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:30:51.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:30:51.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:30:51.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:30:51.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:30:51.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:30:51.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:30:51.906 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:30:51.906 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:30:51.906 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:30:51.906 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-03-09T13:30:51.906 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:30:51.906 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:30:51.906 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-03-09T13:30:51.906 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-03-09T13:30:51.906 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-03-09T13:30:51.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:30:51.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:51.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:30:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836568 2024-03-09T13:30:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836568 2024-03-09T13:30:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568' 2024-03-09T13:30:52.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:52.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:30:52.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673039 2024-03-09T13:30:52.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673039 2024-03-09T13:30:52.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673039' 2024-03-09T13:30:52.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:52.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:30:52.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509516 2024-03-09T13:30:52.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509516 2024-03-09T13:30:52.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673039 2-64424509516' 2024-03-09T13:30:52.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:52.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:30:52.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612624 2024-03-09T13:30:52.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612624 2024-03-09T13:30:52.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673039 2-64424509516 3-313532612624' 2024-03-09T13:30:52.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:52.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:30:52.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182457 2024-03-09T13:30:52.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182457 2024-03-09T13:30:52.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673039 2-64424509516 3-313532612624 4-107374182457' 2024-03-09T13:30:52.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:52.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:30:52.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252871 2024-03-09T13:30:52.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252871 2024-03-09T13:30:52.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673039 2-64424509516 3-313532612624 4-107374182457 5-360777252871' 2024-03-09T13:30:52.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:52.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:30:52.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855405 2024-03-09T13:30:52.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855405 2024-03-09T13:30:52.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673039 2-64424509516 3-313532612624 4-107374182457 5-360777252871 6-150323855405' 2024-03-09T13:30:52.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:52.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:30:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691878 2024-03-09T13:30:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691878 2024-03-09T13:30:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673039 2-64424509516 3-313532612624 4-107374182457 5-360777252871 6-150323855405 7-171798691878' 2024-03-09T13:30:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:52.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:30:52.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528352 2024-03-09T13:30:52.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528352 2024-03-09T13:30:52.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673039 2-64424509516 3-313532612624 4-107374182457 5-360777252871 6-150323855405 7-171798691878 8-193273528352' 2024-03-09T13:30:52.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:30:52.846 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:30:52.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023938 2024-03-09T13:30:52.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023938 2024-03-09T13:30:52.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673039 2-64424509516 3-313532612624 4-107374182457 5-360777252871 6-150323855405 7-171798691878 8-193273528352 9-390842023938' 2024-03-09T13:30:52.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:52.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836568 2024-03-09T13:30:52.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:53.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:30:53.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836568 2024-03-09T13:30:53.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:53.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836568 2024-03-09T13:30:53.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-03-09T13:30:53.002 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836568 2024-03-09T13:30:53.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:30:53.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836568 -lt 21474836568 2024-03-09T13:30:53.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:53.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673039 2024-03-09T13:30:53.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:53.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:30:53.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673039 2024-03-09T13:30:53.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:53.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673039 2024-03-09T13:30:53.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673039' 2024-03-09T13:30:53.279 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673039 2024-03-09T13:30:53.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:30:53.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673040 -lt 42949673039 2024-03-09T13:30:53.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:53.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509516 2024-03-09T13:30:53.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:53.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:30:53.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509516 2024-03-09T13:30:53.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:53.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509516 2024-03-09T13:30:53.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509516' 2024-03-09T13:30:53.557 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509516 2024-03-09T13:30:53.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:30:53.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509516 -lt 64424509516 2024-03-09T13:30:53.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:53.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612624 2024-03-09T13:30:53.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:53.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:30:53.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612624 2024-03-09T13:30:53.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:53.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612624 2024-03-09T13:30:53.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612624' 2024-03-09T13:30:53.863 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 313532612624 2024-03-09T13:30:53.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:30:54.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612624 -lt 313532612624 2024-03-09T13:30:54.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:54.138 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182457 2024-03-09T13:30:54.138 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:54.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:30:54.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182457 2024-03-09T13:30:54.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:54.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182457 2024-03-09T13:30:54.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182457' 2024-03-09T13:30:54.140 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182457 2024-03-09T13:30:54.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:30:54.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182457 -lt 107374182457 2024-03-09T13:30:54.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:54.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252871 2024-03-09T13:30:54.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:54.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:30:54.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252871 2024-03-09T13:30:54.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:54.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252871 2024-03-09T13:30:54.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252871' 2024-03-09T13:30:54.418 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 360777252871 2024-03-09T13:30:54.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:30:54.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252871 -lt 360777252871 2024-03-09T13:30:54.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:54.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855405 2024-03-09T13:30:54.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:54.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:30:54.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855405 2024-03-09T13:30:54.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:54.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855405 2024-03-09T13:30:54.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855405' 2024-03-09T13:30:54.739 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 150323855405 2024-03-09T13:30:54.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:30:55.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855406 -lt 150323855405 2024-03-09T13:30:55.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:55.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691878 2024-03-09T13:30:55.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:55.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:30:55.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691878 2024-03-09T13:30:55.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:55.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691878 2024-03-09T13:30:55.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691878' 2024-03-09T13:30:55.018 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691878 2024-03-09T13:30:55.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:30:55.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691879 -lt 171798691878 2024-03-09T13:30:55.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:55.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528352 2024-03-09T13:30:55.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:55.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:30:55.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528352 2024-03-09T13:30:55.300 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:55.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528352 2024-03-09T13:30:55.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528352' 2024-03-09T13:30:55.302 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528352 2024-03-09T13:30:55.302 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:30:55.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528352 -lt 193273528352 2024-03-09T13:30:55.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:30:55.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-390842023938 2024-03-09T13:30:55.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:30:55.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:30:55.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-390842023938 2024-03-09T13:30:55.581 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:30:55.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023938 2024-03-09T13:30:55.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 390842023938' 2024-03-09T13:30:55.582 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 390842023938 2024-03-09T13:30:55.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:30:55.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023939 -lt 390842023938 2024-03-09T13:30:55.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:30:55.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:30:55.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:30:56.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:30:56.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:30:56.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:30:56.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:30:56.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:30:56.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:30:56.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:30:56.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:30:56.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:30:56.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:30:56.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:30:56.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:30:56.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:30:56.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:30:56.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:30:56.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:30:56.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:30:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-03-09T13:30:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T13:30:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T13:30:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-03-09T13:30:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-03-09T13:30:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T13:30:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:30:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:30:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229557"' 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229557' 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 229559"' 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 229559' 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 229557 229559' 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229557 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229560: /' 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T13:30:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/229562: /' 2024-03-09T13:31:00.141 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:31:00.141 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:31:00.141 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:31:00.142 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:31:00.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:31:00.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:31:00.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:31:00.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:31:00.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T13:31:00.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T13:31:00.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:31:00.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:31:00.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:31:00.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:31:00.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:31:00.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:31:00.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:31:00.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:31:00.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:31:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:31:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:31:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:31:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:31:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:31:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:31:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:31:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:31:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:31:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T13:31:00.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T13:31:00.147 INFO:tasks.workunit.client.0.smithi162.stderr:229562: start osd.9 2024-03-09T13:31:00.147 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:31:00.222 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:31:00.223 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T13:31:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:31:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:31:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:31:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:31:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:31:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:31:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T13:31:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T13:31:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:31:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:31:00.226 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:31:00.226 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:31:00.226 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:31:00.226 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:31:00.226 INFO:tasks.workunit.client.0.smithi162.stderr:229560: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:31:00.226 INFO:tasks.workunit.client.0.smithi162.stderr:229560: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:31:00.226 INFO:tasks.workunit.client.0.smithi162.stderr:229560: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:31:00.226 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:00.226 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:00.226 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:31:00.226 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:31:00.226 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:31:00.226 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:31:00.227 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:31:00.227 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:31:00.227 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:31:00.227 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:31:00.227 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:31:00.227 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T13:31:00.227 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T13:31:00.227 INFO:tasks.workunit.client.0.smithi162.stderr:229560: start osd.5 2024-03-09T13:31:00.227 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:31:04.929 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T13:31:04.929 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T13:31:04.930 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:31:04.930 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:31:04.930 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:31:04.930 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 2024-03-09T13:31:00.173+0000 7f8d5be1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:04.930 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 2024-03-09T13:31:00.175+0000 7f8d5be1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:04.930 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 2024-03-09T13:31:00.176+0000 7f8d5be1e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:04.930 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T13:31:04.930 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:31:04.930 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T13:31:04.930 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:31:04.930 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:31:04.930 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 0 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 2024-03-09T13:31:00.691+0000 7f8d5be1e640 -1 Falling back to public interface 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 1 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:04.931 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 2 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 2024-03-09T13:31:03.306+0000 7f8d5be1e640 -1 osd.9 93 log_to_monitors true 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 3 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229562: osd.9 up in weight 1 up_from 98 up_thru 0 down_at 94 last_c229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T13:31:04.932 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T13:31:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:31:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:31:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:31:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 2024-03-09T13:31:00.252+0000 7f88f5a24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 2024-03-09T13:31:00.254+0000 7f88f5a24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 2024-03-09T13:31:00.255+0000 7f88f5a24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T13:31:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:31:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T13:31:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:31:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:31:04.934 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:04.934 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:31:04.934 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 0 2024-03-09T13:31:04.934 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:04.934 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:04.934 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 2024-03-09T13:31:00.780+0000 7f88f5a24640 -1 Falling back to public interface 2024-03-09T13:31:04.934 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:04.934 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:04.934 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:04.934 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:31:04.934 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 1 2024-03-09T13:31:04.934 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:04.934 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 2 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 2024-03-09T13:31:03.632+0000 7f88f5a24640 -1 osd.5 93 log_to_monitors true 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 3 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:04.935 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:06.140 INFO:tasks.workunit.client.0.smithi162.stderr:229560: osd.5 up in weight 1 up_from 99 up_thru 84 down_at 94 last_clean_interval [84,93) [v2:127.0.0.1:6874/521096859,v1:127.0.0.1:6875/521096859] [v2:127.0.0.1:6876/521096859,v1:127.0.0.1:6877/521096859] exists,up 9fc21c30-bd92-43ad-a6a2-73fee02bea12 2024-03-09T13:31:06.140 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:31:06.140 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:31:06.140 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:31:06.140 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:31:06.141 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 1 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 2 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 3 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 4 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 5 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 6 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 7 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 8 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: 9' 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:06.142 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:31:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836572 2024-03-09T13:31:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836572 2024-03-09T13:31:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572' 2024-03-09T13:31:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:31:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673043 2024-03-09T13:31:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673043 2024-03-09T13:31:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673043' 2024-03-09T13:31:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/stlean_interval [91,93) [v2:127.0.0.1:6842/1467595518,v1:127.0.0.1:6843/1467595518] [v2:127.0.0.1:6844/1467595518,v1:127.0.0.1:6845/1467595518] exists,up 822521c6-f894-4524-bb32-532c8efbf8d4 2024-03-09T13:31:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:31:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:31:06.143 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:31:06.144 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:31:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:31:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:31:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 1 2024-03-09T13:31:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 2 2024-03-09T13:31:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 3 2024-03-09T13:31:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 4 2024-03-09T13:31:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 5 2024-03-09T13:31:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 6 2024-03-09T13:31:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 7 2024-03-09T13:31:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 8 2024-03-09T13:31:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: 9' 2024-03-09T13:31:06.145 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:31:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:31:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836573 2024-03-09T13:31:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836573 2024-03-09T13:31:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573' 2024-03-09T13:31:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:31:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673044 2024-03-09T13:31:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673044 2024-03-09T13:31:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044' 2024-03-09T13:31:06.146 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:07.544 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:31:07.545 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509519 2024-03-09T13:31:07.545 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509519 2024-03-09T13:31:07.545 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509519' 2024-03-09T13:31:07.545 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:07.545 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:31:07.545 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612627 2024-03-09T13:31:07.545 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612627 2024-03-09T13:31:07.545 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509519 3-313532612627' 2024-03-09T13:31:07.545 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:07.545 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:31:07.545 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182461 2024-03-09T13:31:07.545 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182461 2024-03-09T13:31:07.545 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509519 3-313532612627 4-107374182461' 2024-03-09T13:31:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:31:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762307 2024-03-09T13:31:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762307 2024-03-09T13:31:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509519 3-313532612627 4-107374182461 5-425201762307' 2024-03-09T13:31:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:31:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855409 2024-03-09T13:31:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855409 2024-03-09T13:31:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509519 3-313532612627 4-107374182461 5-425201762307 6-150323855409' 2024-03-09T13:31:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:31:07.546 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691882 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691882 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509519 3-313532612627 4-107374182461 5-425201762307 6-150323855409 7-171798691882' 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/cloneandalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509520 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509520 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673043 2-64424509520' 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612628 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612628 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673043 2-64424509520 3-313532612628' 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:07.547 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182462 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182462 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673043 2-64424509520 3-313532612628 4-107374182462' 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673043 2-64424509520 3-313532612628 4-107374182462 5-425201762306' 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855410 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855410 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673043 2-64424509520 3-313532612628 4-107374182462 5-425201762306 6-150323855410' 2024-03-09T13:31:07.548 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:31:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691883 2024-03-09T13:31:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691883 2024-03-09T13:31:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673043 2-64424509520 3-313532612628 4-107374182462 5-425201762306 6-150323855410 7-171798691883' 2024-03-09T13:31:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:31:08.268 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528357 2024-03-09T13:31:08.268 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528357 2024-03-09T13:31:08.268 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673043 2-64424509520 3-313532612628 4-107374182462 5-425201762306 6-150323855410 7-171798691883 8-193273528357' 2024-03-09T13:31:08.269 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:08.269 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:31:08.269 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795010 2024-03-09T13:31:08.269 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795010 2024-03-09T13:31:08.269 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673043 2-64424509520 3-313532612628 4-107374182462 5-425201762306 6-150323855410 7-171798691883 8-193273528357 9-420906795010' 2024-03-09T13:31:08.269 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:08.269 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836572 2024-03-09T13:31:08.269 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:08.269 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:31:08.269 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836572 2024-03-09T13:31:08.269 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:08.269 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836572 2024-03-09T13:31:08.269 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-03-09T13:31:08.270 INFO:tasks.workunit.client.0.smithi162.stderr:229560: waiting osd.0 seq 21474836572 2024-03-09T13:31:08.270 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:31:08.270 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836573 -lt 21474836572 2024-03-09T13:31:08.270 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:08.270 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673043 2024-03-09T13:31:08.270 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:08.270 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:31:08.270 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673043 2024-03-09T13:31:08.270 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:08.270 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673043 2024-03-09T13:31:08.270 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-03-09T13:31:08.270 INFO:tasks.workunit.client.0.smithi162.stderr:229560: waiting osd.1 seq 42949673043 2024-03-09T13:31:08.271 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:31:08.271 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673044 -lt 42949673043 2024-03-09T13:31:08.271 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:08.271 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509520 2024-03-09T13:31:08.271 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:08.271 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:31:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.clie.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528356 2024-03-09T13:31:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528356 2024-03-09T13:31:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509519 3-313532612627 4-107374182461 5-425201762307 6-150323855409 7-171798691882 8-193273528356' 2024-03-09T13:31:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:31:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795011 2024-03-09T13:31:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795011 2024-03-09T13:31:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673044 2-64424509519 3-313532612627 4-107374182461 5-425201762307 6-150323855409 7-171798691882 8-193273528356 9-420906795011' 2024-03-09T13:31:08.278 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836573 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836573 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836573 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: waiting osd.0 seq 21474836573 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836573 -lt 21474836573 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673044 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:31:08.279 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673044 2024-03-09T13:31:08.280 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:08.280 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673044 2024-03-09T13:31:08.280 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673044' 2024-03-09T13:31:08.280 INFO:tasks.workunit.client.0.smithi162.stderr:229562: waiting osd.1 seq 42949673044 2024-03-09T13:31:08.280 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:31:08.280 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673044 -lt 42949673044 2024-03-09T13:31:08.280 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:08.280 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509519 2024-03-09T13:31:08.280 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:08.280 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:31:09.184 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.cnt.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509520 2024-03-09T13:31:09.184 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:09.184 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509520 2024-03-09T13:31:09.184 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-03-09T13:31:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:229560: waiting osd.2 seq 64424509520 2024-03-09T13:31:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:31:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509520 -lt 64424509520 2024-03-09T13:31:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612628 2024-03-09T13:31:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:31:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612628 2024-03-09T13:31:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612628 2024-03-09T13:31:09.185 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612628' 2024-03-09T13:31:09.186 INFO:tasks.workunit.client.0.smithi162.stderr:229560: waiting osd.3 seq 313532612628 2024-03-09T13:31:09.186 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:31:09.186 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612629 -lt 313532612628 2024-03-09T13:31:09.186 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:09.186 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182462 2024-03-09T13:31:09.186 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:09.186 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:31:09.186 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182462 2024-03-09T13:31:09.186 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:09.186 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182462 2024-03-09T13:31:09.186 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182462' 2024-03-09T13:31:09.186 INFO:tasks.workunit.client.0.smithi162.stderr:229560: waiting osd.4 seq 107374182462 2024-03-09T13:31:09.186 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:31:09.187 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182462 -lt 107374182462 2024-03-09T13:31:09.187 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:09.187 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-425201762306 2024-03-09T13:31:09.187 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:09.187 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:31:09.187 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-425201762306 2024-03-09T13:31:09.187 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:09.187 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-03-09T13:31:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting lient.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509519 2024-03-09T13:31:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509519 2024-03-09T13:31:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509519' 2024-03-09T13:31:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:229562: waiting osd.2 seq 64424509519 2024-03-09T13:31:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:31:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509520 -lt 64424509519 2024-03-09T13:31:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612627 2024-03-09T13:31:09.231 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:09.232 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:31:09.232 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612627 2024-03-09T13:31:09.232 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:09.232 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612627 2024-03-09T13:31:09.232 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612627' 2024-03-09T13:31:09.232 INFO:tasks.workunit.client.0.smithi162.stderr:229562: waiting osd.3 seq 313532612627 2024-03-09T13:31:09.232 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:31:09.232 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612629 -lt 313532612627 2024-03-09T13:31:09.232 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:09.232 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182461 2024-03-09T13:31:09.232 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:09.232 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:31:09.232 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182461 2024-03-09T13:31:09.233 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:09.233 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182461 2024-03-09T13:31:09.233 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182461' 2024-03-09T13:31:09.233 INFO:tasks.workunit.client.0.smithi162.stderr:229562: waiting osd.4 seq 107374182461 2024-03-09T13:31:09.233 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:31:09.233 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182462 -lt 107374182461 2024-03-09T13:31:09.233 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:09.233 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-425201762307 2024-03-09T13:31:09.233 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:09.233 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:31:09.233 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-425201762307 2024-03-09T13:31:09.233 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:09.234 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762307 2024-03-09T13:31:10.408 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitiosd.5 seq 425201762306' 2024-03-09T13:31:10.408 INFO:tasks.workunit.client.0.smithi162.stderr:229560: waiting osd.5 seq 425201762306 2024-03-09T13:31:10.408 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:31:10.408 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762307 -lt 425201762306 2024-03-09T13:31:10.408 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855410 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855410 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855410 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: waiting osd.6 seq 150323855410 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855410 -lt 150323855410 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691883 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:10.409 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691883 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691883 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691883' 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: waiting osd.7 seq 171798691883 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691884 -lt 171798691883 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528357 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528357 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:10.410 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528357 2024-03-09T13:31:10.411 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528357' 2024-03-09T13:31:10.411 INFO:tasks.workunit.client.0.smithi162.stderr:229560: waiting osd.8 seq 193273528357 2024-03-09T13:31:10.411 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:31:10.411 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528357 -lt 193273528357 2024-03-09T13:31:10.449 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stng osd.5 seq 425201762307' 2024-03-09T13:31:10.449 INFO:tasks.workunit.client.0.smithi162.stderr:229562: waiting osd.5 seq 425201762307 2024-03-09T13:31:10.449 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:31:10.449 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762307 -lt 425201762307 2024-03-09T13:31:10.449 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:10.449 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855409 2024-03-09T13:31:10.449 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:10.449 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:31:10.449 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855409 2024-03-09T13:31:10.449 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:10.449 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855409 2024-03-09T13:31:10.450 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855409' 2024-03-09T13:31:10.450 INFO:tasks.workunit.client.0.smithi162.stderr:229562: waiting osd.6 seq 150323855409 2024-03-09T13:31:10.450 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:31:10.450 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855410 -lt 150323855409 2024-03-09T13:31:10.450 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:10.450 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691882 2024-03-09T13:31:10.450 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:10.450 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:31:10.450 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691882 2024-03-09T13:31:10.450 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:10.450 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691882 2024-03-09T13:31:10.450 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691882' 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: waiting osd.7 seq 171798691882 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691884 -lt 171798691882 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528356 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528356 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528356 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528356' 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: waiting osd.8 seq 193273528356 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:31:10.451 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528357 -lt 193273528356 2024-03-09T13:31:11.875 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pgats: for s in $seqs 2024-03-09T13:31:11.875 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-420906795010 2024-03-09T13:31:11.875 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:11.875 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:31:11.875 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-420906795010 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795010 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 420906795010' 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: waiting osd.9 seq 420906795010 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795012 -lt 420906795010 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:31:11.876 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:31:11.877 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:31:11.878 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:31:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu_stats: for s in $seqs 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-420906795011 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-420906795011 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795011 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 420906795011' 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: waiting osd.9 seq 420906795011 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795012 -lt 420906795011 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T13:31:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:31:11.976 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T13:31:11.976 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:31:11.976 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:31:11.976 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:31:11.976 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:31:11.976 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:31:13.067 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:31:13.067 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:31:13.067 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:31:13.067 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:31:13.067 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:31:13.067 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:31:13.067 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:31:13.067 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T13:31:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T13:31:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:31:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:31:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:31:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:31:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:31:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:31:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:31:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:31:13.068 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:31:13.069 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:31:13.069 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:31:13.069 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:31:13.069 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:31:13.069 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:31:13.069 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:31:13.069 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:31:13.069 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:31:13.069 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:31:13.185 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgsntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:31:13.185 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:31:13.185 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:31:13.185 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:31:13.185 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:31:13.185 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:31:13.185 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:31:13.185 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:13.186 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:13.186 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T13:31:13.186 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T13:31:13.186 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:31:13.186 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:31:13.186 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:31:13.186 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:31:13.186 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:31:13.186 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:31:13.186 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:31:13.187 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:31:13.187 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:31:13.187 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:31:13.187 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:31:13.187 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:31:13.187 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:31:13.187 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:31:13.188 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:31:13.188 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:31:13.188 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:31:13.188 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:31:13.711 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump 2024-03-09T13:31:13.711 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:31:13.711 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:31:13.711 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:31:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:229560: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:31:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:31:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:229560: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:31:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:31:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:31:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 229559 2024-03-09T13:31:13.852 INFO:tasks.workunit.client.0.smithi162.stderr:pgs 2024-03-09T13:31:13.852 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:31:13.852 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:31:13.852 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:31:13.852 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:13.852 INFO:tasks.workunit.client.0.smithi162.stderr:229562: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:13.852 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:31:13.853 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:31:13.853 INFO:tasks.workunit.client.0.smithi162.stderr:229562: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:31:13.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:31:13.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:31:13.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:31:13.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:31:13.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T13:31:13.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:31:13.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T13:31:13.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:31:13.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:31:13.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:31:13.853 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:31:14.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T13:31:14.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T13:31:14.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:31:14.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:31:14.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:31:14.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:31:14.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:31:14.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:31:14.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:30:38.546280+0000 2024-03-09T13:31:14.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:31:14.675 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:31:14.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:30:38.546280+0000 2024-03-09T13:31:14.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:31:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:30:38.546280+0000 2024-03-09T13:31:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:31:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:31:14.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:31:14.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:31:14.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:31:14.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:31:14.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:31:14.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:31:14.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:30:38.546280+0000 '>' 2024-03-09T13:30:38.546280+0000 2024-03-09T13:31:14.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:31:15.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:31:15.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:31:15.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:31:15.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:31:15.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:31:15.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:31:15.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:31:16.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:30:38.546280+0000 '>' 2024-03-09T13:30:38.546280+0000 2024-03-09T13:31:16.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:31:17.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:31:17.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:31:17.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:31:17.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:31:17.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:31:17.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:31:17.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:31:17.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:30:38.546280+0000 '>' 2024-03-09T13:30:38.546280+0000 2024-03-09T13:31:17.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:31:18.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:31:18.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:31:18.546 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:31:18.546 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:31:18.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:31:18.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:31:18.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:31:18.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:31:15.371099+0000 '>' 2024-03-09T13:30:38.546280+0000 2024-03-09T13:31:18.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:31:18.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T13:31:18.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:31:18.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:31:18.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:31:18.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 233182"' 2024-03-09T13:31:18.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 233182' 2024-03-09T13:31:18.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:31:18.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:31:18.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:31:18.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 233184"' 2024-03-09T13:31:18.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:31:18.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 233184' 2024-03-09T13:31:18.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T13:31:18.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 233182 233184' 2024-03-09T13:31:18.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/233185: /' 2024-03-09T13:31:18.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:31:18.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:31:18.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 233182 2024-03-09T13:31:18.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:31:18.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/233187: /' 2024-03-09T13:31:21.385 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:31:21.385 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:31:21.385 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T13:31:21.386 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:31:21.386 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:31:21.386 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:31:21.386 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:31:21.386 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T13:31:21.386 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:31:21.386 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T13:31:21.386 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:31:21.386 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:31:21.386 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:31:21.386 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:31:21.386 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: _ 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: hinfo_key 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: snapset 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:31:21.387 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:31:21.388 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T13:31:21.388 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:31:21.388 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:31:21.388 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:31:21.388 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:31:21.388 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:31:21.388 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:31:21.388 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:31:21.389 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:31:21.390 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:31:21.390 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:31:21.390 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:31:21.390 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:31:21.390 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:31:21.390 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:31:21.390 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T13:31:21.390 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T13:31:21.390 INFO:tasks.workunit.client.0.smithi162.stderr:233187: start osd.9 2024-03-09T13:31:21.525 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:31:21.525 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:31:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T13:31:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:31:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:31:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:31:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:31:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T13:31:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:31:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T13:31:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:31:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:31:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:31:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:31:21.526 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: _ 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: hinfo_key 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: snapset 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:31:21.527 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:31:21.528 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T13:31:21.528 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:31:21.528 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:31:21.528 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:31:21.528 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:31:21.528 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:31:21.528 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:31:21.528 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T13:31:21.528 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T13:31:21.529 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:31:21.529 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:31:21.529 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:31:21.529 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:31:21.529 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:31:21.529 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:31:21.529 INFO:tasks.workunit.client.0.smithi162.stderr:233185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:31:21.529 INFO:tasks.workunit.client.0.smithi162.stderr:233185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:31:21.529 INFO:tasks.workunit.client.0.smithi162.stderr:233185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:31:21.529 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:21.529 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:21.530 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:31:21.530 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:31:21.530 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:31:21.530 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:31:21.530 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:31:21.530 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:31:21.530 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:31:21.530 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:31:21.530 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:31:21.530 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T13:31:21.530 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T13:31:21.530 INFO:tasks.workunit.client.0.smithi162.stderr:233185: start osd.5 2024-03-09T13:31:25.996 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:31:25.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T13:31:25.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T13:31:25.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:31:25.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:31:25.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:31:25.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 2024-03-09T13:31:21.418+0000 7f30e876d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:25.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 2024-03-09T13:31:21.421+0000 7f30e876d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 2024-03-09T13:31:21.422+0000 7f30e876d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 0 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 2024-03-09T13:31:21.942+0000 7f30e876d640 -1 Falling back to public interface 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:25.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:31:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 1 2024-03-09T13:31:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:31:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:31:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 2 2024-03-09T13:31:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:31:25.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 2024-03-09T13:31:24.555+0000 7f30e876d640 -1 osd.9 102 log_to_monitors true 2024-03-09T13:31:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:31:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 3 2024-03-09T13:31:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:25.999 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:31:26.258 INFO:tasks.workunit.client.0.smithi162.stderr:233187: osd.9 up in weight 1 up_from 107 up_thru 0 down_overy_ops 2024-03-09T13:31:26.258 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T13:31:26.258 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T13:31:26.258 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:31:26.258 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:31:26.258 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:31:26.258 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 2024-03-09T13:31:21.554+0000 7f78f6124640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:26.258 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 2024-03-09T13:31:21.556+0000 7f78f6124640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 2024-03-09T13:31:21.557+0000 7f78f6124640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 0 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 2024-03-09T13:31:22.076+0000 7f78f6124640 -1 Falling back to public interface 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:26.259 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 1 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 2 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 2024-03-09T13:31:24.816+0000 7f78f6124640 -1 osd.5 102 log_to_monitors true 2024-03-09T13:31:26.260 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:26.261 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:26.261 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:26.261 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:31:26.261 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 3 2024-03-09T13:31:26.261 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:26.261 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:26.996 INFO:tasks.workunit.client.0.smithi162.stderr:233185: osd.5 up in weight 1 up_from 107 up_thru 107 dowat 103 last_clean_interval [98,102) [v2:127.0.0.1:6842/1044895232,v1:127.0.0.1:6843/1044895232] [v2:127.0.0.1:6844/1044895232,v1:127.0.0.1:6845/1044895232] exists,up 822521c6-f894-4524-bb32-532c8efbf8d4 2024-03-09T13:31:26.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:31:26.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:31:26.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:31:26.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:31:26.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:31:26.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:31:26.996 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:31:26.997 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 1 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 2 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 3 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 4 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 5 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 6 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 7 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 8 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: 9' 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836578 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836578 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578' 2024-03-09T13:31:26.998 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:26.999 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:31:26.999 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673049 2024-03-09T13:31:26.999 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673049 2024-03-09T13:31:26.999 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673049' 2024-03-09T13:31:26.999 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:27.404 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clon_at 103 last_clean_interval [99,102) [v2:127.0.0.1:6874/1913113804,v1:127.0.0.1:6875/1913113804] [v2:127.0.0.1:6876/1913113804,v1:127.0.0.1:6877/1913113804] exists,up 9fc21c30-bd92-43ad-a6a2-73fee02bea12 2024-03-09T13:31:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:31:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:31:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:31:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:31:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:31:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:31:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:31:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:31:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:233185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:31:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:233185: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:31:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:31:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:31:27.405 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:31:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:31:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:31:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:31:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:31:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:31:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:31:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:31:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 1 2024-03-09T13:31:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 2 2024-03-09T13:31:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 3 2024-03-09T13:31:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 4 2024-03-09T13:31:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 5 2024-03-09T13:31:27.406 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 6 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 7 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 8 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: 9' 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836579 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836579 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579' 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673050 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673050 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673050' 2024-03-09T13:31:27.407 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:28.245 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/cne.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:31:28.246 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509526 2024-03-09T13:31:28.246 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509526 2024-03-09T13:31:28.246 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509526' 2024-03-09T13:31:28.246 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:28.246 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:31:28.246 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612634 2024-03-09T13:31:28.247 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612634 2024-03-09T13:31:28.247 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509526 3-313532612634' 2024-03-09T13:31:28.247 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:28.247 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:31:28.247 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182467 2024-03-09T13:31:28.247 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182467 2024-03-09T13:31:28.248 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509526 3-313532612634 4-107374182467' 2024-03-09T13:31:28.248 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:28.248 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:31:28.248 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500674 2024-03-09T13:31:28.248 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500674 2024-03-09T13:31:28.248 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509526 3-313532612634 4-107374182467 5-459561500674' 2024-03-09T13:31:28.248 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:28.249 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:31:28.249 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855415 2024-03-09T13:31:28.249 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855415 2024-03-09T13:31:28.249 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509526 3-313532612634 4-107374182467 5-459561500674 6-150323855415' 2024-03-09T13:31:28.249 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:28.249 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:31:28.250 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691889 2024-03-09T13:31:28.250 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691889 2024-03-09T13:31:28.250 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509526 3-313532612634 4-107374182467 5-459561500674 6-150323855415 7-171798691889' 2024-03-09T13:31:28.250 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:28.250 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:31:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/lone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:31:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509527 2024-03-09T13:31:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509527 2024-03-09T13:31:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673050 2-64424509527' 2024-03-09T13:31:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:31:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612635 2024-03-09T13:31:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612635 2024-03-09T13:31:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673050 2-64424509527 3-313532612635' 2024-03-09T13:31:28.374 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182468 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182468 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673050 2-64424509527 3-313532612635 4-107374182468' 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500675 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500675 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673050 2-64424509527 3-313532612635 4-107374182468 5-459561500675' 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855416 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855416 2024-03-09T13:31:28.375 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673050 2-64424509527 3-313532612635 4-107374182468 5-459561500675 6-150323855416' 2024-03-09T13:31:28.376 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:28.376 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:31:28.376 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691890 2024-03-09T13:31:28.376 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691890 2024-03-09T13:31:28.376 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673050 2-64424509527 3-313532612635 4-107374182468 5-459561500675 6-150323855416 7-171798691890' 2024-03-09T13:31:28.376 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:28.376 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:31:30.262 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntcephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528362 2024-03-09T13:31:30.262 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528362 2024-03-09T13:31:30.262 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509526 3-313532612634 4-107374182467 5-459561500674 6-150323855415 7-171798691889 8-193273528362' 2024-03-09T13:31:30.262 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:30.262 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500674 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500674 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673049 2-64424509526 3-313532612634 4-107374182467 5-459561500674 6-150323855415 7-171798691889 8-193273528362 9-459561500674' 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836578 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836578 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836578 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: waiting osd.0 seq 21474836578 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:31:30.263 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836577 -lt 21474836578 2024-03-09T13:31:30.264 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:31:30.264 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:31:30.264 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:31:30.264 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836578 2024-03-09T13:31:30.264 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:30.264 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673049 2024-03-09T13:31:30.264 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:30.264 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:31:30.264 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673049 2024-03-09T13:31:30.264 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:30.265 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673049 2024-03-09T13:31:30.265 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673049' 2024-03-09T13:31:30.265 INFO:tasks.workunit.client.0.smithi162.stderr:233187: waiting osd.1 seq 42949673049 2024-03-09T13:31:30.265 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:31:30.450 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673051 -lt 4294967u/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528363 2024-03-09T13:31:30.450 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528363 2024-03-09T13:31:30.450 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673050 2-64424509527 3-313532612635 4-107374182468 5-459561500675 6-150323855416 7-171798691890 8-193273528363' 2024-03-09T13:31:30.450 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:30.450 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:31:30.451 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500675 2024-03-09T13:31:30.451 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500675 2024-03-09T13:31:30.451 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673050 2-64424509527 3-313532612635 4-107374182468 5-459561500675 6-150323855416 7-171798691890 8-193273528363 9-459561500675' 2024-03-09T13:31:30.451 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:30.451 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836579 2024-03-09T13:31:30.451 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:30.452 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:31:30.452 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836579 2024-03-09T13:31:30.452 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:30.452 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836579 2024-03-09T13:31:30.452 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-03-09T13:31:30.452 INFO:tasks.workunit.client.0.smithi162.stderr:233185: waiting osd.0 seq 21474836579 2024-03-09T13:31:30.452 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:31:30.453 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836577 -lt 21474836579 2024-03-09T13:31:30.453 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:31:30.453 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:31:30.453 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:31:30.453 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836579 2024-03-09T13:31:30.453 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:30.453 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673050 2024-03-09T13:31:30.453 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:30.454 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:31:30.454 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673050 2024-03-09T13:31:30.454 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:30.454 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673050 2024-03-09T13:31:30.454 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-03-09T13:31:30.454 INFO:tasks.workunit.client.0.smithi162.stderr:233185: waiting osd.1 seq 42949673050 2024-03-09T13:31:30.454 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:31:31.230 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673051 -lt 429493049 2024-03-09T13:31:31.230 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:31.230 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509526 2024-03-09T13:31:31.230 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:31.230 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:31:31.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509526 2024-03-09T13:31:31.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:31.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509526 2024-03-09T13:31:31.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-03-09T13:31:31.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: waiting osd.2 seq 64424509526 2024-03-09T13:31:31.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:31:31.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509527 -lt 64424509526 2024-03-09T13:31:31.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:31.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612634 2024-03-09T13:31:31.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:31.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:31:31.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612634 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612634 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612634' 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: waiting osd.3 seq 313532612634 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612635 -lt 313532612634 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182467 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182467 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182467 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182467' 2024-03-09T13:31:31.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: waiting osd.4 seq 107374182467 2024-03-09T13:31:31.233 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:31:31.233 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182469 -lt 107374182467 2024-03-09T13:31:31.233 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:31.233 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-459561500674 2024-03-09T13:31:31.233 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:31.369 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpe673050 2024-03-09T13:31:31.370 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:31.370 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509527 2024-03-09T13:31:31.370 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:31.370 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:31:31.370 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509527 2024-03-09T13:31:31.370 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:31.370 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509527 2024-03-09T13:31:31.371 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-03-09T13:31:31.371 INFO:tasks.workunit.client.0.smithi162.stderr:233185: waiting osd.2 seq 64424509527 2024-03-09T13:31:31.371 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:31:31.371 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509527 -lt 64424509527 2024-03-09T13:31:31.371 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:31.371 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612635 2024-03-09T13:31:31.371 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:31.372 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:31:31.372 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612635 2024-03-09T13:31:31.372 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:31.372 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612635 2024-03-09T13:31:31.372 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612635' 2024-03-09T13:31:31.372 INFO:tasks.workunit.client.0.smithi162.stderr:233185: waiting osd.3 seq 313532612635 2024-03-09T13:31:31.372 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:31:31.372 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612635 -lt 313532612635 2024-03-09T13:31:31.372 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:31.372 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182468 2024-03-09T13:31:31.373 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:31.373 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:31:31.373 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182468 2024-03-09T13:31:31.373 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:31.373 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182468 2024-03-09T13:31:31.373 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182468' 2024-03-09T13:31:31.373 INFO:tasks.workunit.client.0.smithi162.stderr:233185: waiting osd.4 seq 107374182468 2024-03-09T13:31:31.373 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:31:31.373 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182469 -lt 107374182468 2024-03-09T13:31:31.373 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:31.373 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-459561500675 2024-03-09T13:31:31.374 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:32.230 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helrs.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:31:32.230 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-459561500674 2024-03-09T13:31:32.230 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:32.230 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500674 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 459561500674' 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: waiting osd.5 seq 459561500674 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500676 -lt 459561500674 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855415 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855415 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855415 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855415' 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: waiting osd.6 seq 150323855415 2024-03-09T13:31:32.231 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855417 -lt 150323855415 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691889 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691889 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691889 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691889' 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: waiting osd.7 seq 171798691889 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691890 -lt 171798691889 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:32.232 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528362 2024-03-09T13:31:32.233 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:32.233 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:31:32.233 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528362 2024-03-09T13:31:32.233 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:32.233 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528362 2024-03-09T13:31:32.313 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/pers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:31:32.313 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-459561500675 2024-03-09T13:31:32.313 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:32.313 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500675 2024-03-09T13:31:32.313 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 459561500675' 2024-03-09T13:31:32.313 INFO:tasks.workunit.client.0.smithi162.stderr:233185: waiting osd.5 seq 459561500675 2024-03-09T13:31:32.313 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:31:32.313 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500676 -lt 459561500675 2024-03-09T13:31:32.313 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:32.313 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855416 2024-03-09T13:31:32.314 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:32.314 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:31:32.314 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855416 2024-03-09T13:31:32.314 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:32.314 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855416 2024-03-09T13:31:32.314 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855416' 2024-03-09T13:31:32.314 INFO:tasks.workunit.client.0.smithi162.stderr:233185: waiting osd.6 seq 150323855416 2024-03-09T13:31:32.314 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:31:32.314 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855417 -lt 150323855416 2024-03-09T13:31:32.314 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:32.314 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691890 2024-03-09T13:31:32.314 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:32.314 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691890 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691890 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691890' 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: waiting osd.7 seq 171798691890 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691890 -lt 171798691890 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528363 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528363 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:32.315 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528363 2024-03-09T13:31:33.910 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtesclone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528362' 2024-03-09T13:31:33.910 INFO:tasks.workunit.client.0.smithi162.stderr:233187: waiting osd.8 seq 193273528362 2024-03-09T13:31:33.910 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:31:33.911 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528363 -lt 193273528362 2024-03-09T13:31:33.911 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:33.911 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500674 2024-03-09T13:31:33.911 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:33.911 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:31:33.911 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500674 2024-03-09T13:31:33.911 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:33.912 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500674 2024-03-09T13:31:33.912 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500674' 2024-03-09T13:31:33.912 INFO:tasks.workunit.client.0.smithi162.stderr:233187: waiting osd.9 seq 459561500674 2024-03-09T13:31:33.912 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:31:33.912 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500676 -lt 459561500674 2024-03-09T13:31:33.912 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:31:33.913 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:33.913 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:33.913 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:31:33.913 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:31:33.913 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:31:33.913 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:31:33.914 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:31:33.914 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:31:33.914 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:31:33.914 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:31:33.914 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:31:33.915 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:31:33.915 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:33.915 INFO:tasks.workunit.client.0.smithi162.stderr:233187: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:33.915 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:31:33.915 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:31:33.915 INFO:tasks.workunit.client.0.smithi162.stderr:233187: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:31:33.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:31:34.034 INFO:tasks.workunit.client.0.smithi162.stderr:t/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528363' 2024-03-09T13:31:34.034 INFO:tasks.workunit.client.0.smithi162.stderr:233185: waiting osd.8 seq 193273528363 2024-03-09T13:31:34.034 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:31:34.034 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528363 -lt 193273528363 2024-03-09T13:31:34.034 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:34.034 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500675 2024-03-09T13:31:34.034 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:34.034 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:31:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500675 2024-03-09T13:31:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500675 2024-03-09T13:31:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500675' 2024-03-09T13:31:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:233185: waiting osd.9 seq 459561500675 2024-03-09T13:31:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:31:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500676 -lt 459561500675 2024-03-09T13:31:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:31:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:31:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:31:34.035 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:233185: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:233185: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:31:34.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:31:34.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 233184 2024-03-09T13:31:34.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:31:34.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:31:34.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:31:34.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T13:31:34.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:31:34.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:31:34.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:31:34.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-03-09T13:31:34.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T13:31:34.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T13:31:34.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-09T13:31:34.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-03-09T13:31:34.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T13:31:34.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:31:34.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:31:34.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:31:34.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 236461"' 2024-03-09T13:31:34.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 236461' 2024-03-09T13:31:34.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:31:34.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:31:34.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:31:34.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 236463"' 2024-03-09T13:31:34.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:31:34.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 236463' 2024-03-09T13:31:34.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T13:31:34.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 236461 236463' 2024-03-09T13:31:34.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:31:34.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:31:34.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 236461 2024-03-09T13:31:34.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/236464: /' 2024-03-09T13:31:34.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:31:34.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/236466: /' 2024-03-09T13:31:36.861 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:31:36.861 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:31:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T13:31:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:31:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:31:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:31:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:31:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T13:31:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:31:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T13:31:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:31:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:31:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:31:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:31:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:31:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:31:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:31:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:31:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:31:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T13:31:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:31:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:31:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-09T13:31:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:236466: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:31:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T13:31:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:31:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:31:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T13:31:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:31:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:31:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:31:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:31:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:31:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:31:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T13:31:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T13:31:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:31:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:31:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:31:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:31:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:31:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:31:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:236466: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:31:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T13:31:36.867 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T13:31:36.867 INFO:tasks.workunit.client.0.smithi162.stderr:236466: start osd.5 2024-03-09T13:31:36.867 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:31:36.948 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:31:36.949 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:31:36.949 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:31:36.949 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:31:36.949 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:31:36.949 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:31:36.949 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:31:36.949 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:31:36.949 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:31:36.949 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:31:36.949 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:31:36.949 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:31:36.950 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:31:36.950 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:31:36.950 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:31:36.950 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:31:36.950 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:31:36.950 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:31:36.950 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:31:36.950 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:31:36.950 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:31:36.950 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:31:36.950 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T13:31:36.950 INFO:tasks.workunit.client.0.smithi162.stderr:236464: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T13:31:36.951 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:31:36.951 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:31:36.951 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:31:36.951 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:31:36.951 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:31:36.951 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:31:36.951 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:31:36.951 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:31:36.951 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:36.952 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:31:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:31:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:31:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:31:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:31:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:31:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:31:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:31:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:31:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:31:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:31:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:236464: start osd.3 2024-03-09T13:31:36.953 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:31:41.659 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T13:31:41.659 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T13:31:41.659 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:31:41.659 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:31:41.659 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:31:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 2024-03-09T13:31:36.893+0000 7fb88b822640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 2024-03-09T13:31:36.907+0000 7fb88b822640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 2024-03-09T13:31:36.908+0000 7fb88b822640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T13:31:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:31:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T13:31:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:31:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:31:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:31:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 0 2024-03-09T13:31:41.660 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 2024-03-09T13:31:37.430+0000 7fb88b822640 -1 Falling back to public interface 2024-03-09T13:31:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:31:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 1 2024-03-09T13:31:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:41.661 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:31:41.662 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 2 2024-03-09T13:31:41.662 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:41.662 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:41.662 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 2024-03-09T13:31:40.083+0000 7fb88b822640 -1 osd.5 109 log_to_monitors true 2024-03-09T13:31:41.662 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:41.662 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:41.662 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:41.662 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:31:41.662 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 3 2024-03-09T13:31:41.662 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:41.662 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:236466: osd.5 up in weight 1 up_from 113 up_thru 113 down_at 110 l236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:31:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:31:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:31:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:31:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:31:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 2024-03-09T13:31:36.978+0000 7f4eaa8d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 2024-03-09T13:31:36.984+0000 7f4eaa8d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 2024-03-09T13:31:36.985+0000 7f4eaa8d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:41.665 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:31:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:31:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:31:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:31:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:31:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:31:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 0 2024-03-09T13:31:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:31:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 2024-03-09T13:31:37.508+0000 7f4eaa8d9640 -1 Falling back to public interface 2024-03-09T13:31:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:41.666 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:41.667 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:41.667 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:31:41.667 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 1 2024-03-09T13:31:41.667 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:41.667 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:31:41.667 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:41.667 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:41.667 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:41.667 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:31:41.667 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 2 2024-03-09T13:31:41.667 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:41.667 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:31:41.668 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:41.668 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 2024-03-09T13:31:40.284+0000 7f4eaa8d9640 -1 osd.3 109 log_to_monitors true 2024-03-09T13:31:41.668 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:41.668 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:41.668 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:31:41.668 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 3 2024-03-09T13:31:41.668 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:41.668 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:31:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:236464: osd.3 up in weight 1 up_from 114 up_thru 114 down_at 110 last_clean_interval [73,109) [v2:127.0.0.1:6874/539222183,v1:127.0.0.1:6875/539222183] [v2:127.0.0.1:6876/539222183,v1:127.0.0.1:6877/539222183] exists,up e89c8cb7-e250-4035-b64e-470d42b71834 2024-03-09T13:31:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:31:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:31:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:31:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:31:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:31:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:31:42.297 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:31:42.298 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 1 2024-03-09T13:31:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 2 2024-03-09T13:31:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 3 2024-03-09T13:31:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 4 2024-03-09T13:31:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 5 2024-03-09T13:31:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 6 2024-03-09T13:31:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 7 2024-03-09T13:31:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 8 2024-03-09T13:31:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:236464: 9' 2024-03-09T13:31:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:31:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:31:42.299 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836584 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836584 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584' 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673054 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673054 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673054' 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/ast_clean_interval [107,109) [v2:127.0.0.1:6826/3084039163,v1:127.0.0.1:6827/3084039163] [v2:127.0.0.1:6828/3084039163,v1:127.0.0.1:6829/3084039163] exists,up 9fc21c30-bd92-43ad-a6a2-73fee02bea12 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:31:42.300 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:31:42.301 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:31:42.301 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:31:42.301 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:31:42.301 INFO:tasks.workunit.client.0.smithi162.stderr:236466: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:31:42.301 INFO:tasks.workunit.client.0.smithi162.stderr:236466: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:31:42.301 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 1 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 2 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 3 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 4 2024-03-09T13:31:42.302 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 5 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 6 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 7 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 8 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: 9' 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836583 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836583 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583' 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673055 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673055 2024-03-09T13:31:42.303 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673055' 2024-03-09T13:31:42.304 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.142 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:31:43.142 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509532 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509532 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673055 2-64424509532' 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271746 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271746 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673055 2-64424509532 3-489626271746' 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182472 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182472 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673055 2-64424509532 3-489626271746 4-107374182472' 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.143 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304451 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304451 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673055 2-64424509532 3-489626271746 4-107374182472 5-485331304451' 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855420 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855420 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673055 2-64424509532 3-489626271746 4-107374182472 5-485331304451 6-150323855420' 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691894 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691894 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673055 2-64424509532 3-489626271746 4-107374182472 5-485331304451 6-150323855420 7-171798691894' 2024-03-09T13:31:43.144 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtesqa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509531 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509531 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509531' 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271747 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271747 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509531 3-489626271747' 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182473 2024-03-09T13:31:43.145 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182473 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509531 3-489626271747 4-107374182473' 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=485331304450 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 485331304450 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509531 3-489626271747 4-107374182473 5-485331304450' 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855421 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855421 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509531 3-489626271747 4-107374182473 5-485331304450 6-150323855421' 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691895 2024-03-09T13:31:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691895 2024-03-09T13:31:43.147 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509531 3-489626271747 4-107374182473 5-485331304450 6-150323855421 7-171798691895' 2024-03-09T13:31:43.147 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.147 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:31:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528368 2024-03-09T13:31:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528368 2024-03-09T13:31:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509531 3-489626271747 4-107374182473 5-485331304450 6-150323855421 7-171798691895 8-193273528368' 2024-03-09T13:31:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:31:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500680 2024-03-09T13:31:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500680 2024-03-09T13:31:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673054 2-64424509531 3-489626271747 4-107374182473 5-485331304450 6-150323855421 7-171798691895 8-193273528368 9-459561500680' 2024-03-09T13:31:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836584 2024-03-09T13:31:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:31:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836584 2024-03-09T13:31:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836584 2024-03-09T13:31:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-03-09T13:31:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:236464: waiting osd.0 seq 21474836584 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836584 -lt 21474836584 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673054 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673054 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673054 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673054' 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: waiting osd.1 seq 42949673054 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673056 -lt 42949673054 2024-03-09T13:31:43.868 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509531 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clonet/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528367 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528367 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673055 2-64424509532 3-489626271746 4-107374182472 5-485331304451 6-150323855420 7-171798691894 8-193273528367' 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500679 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500679 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836583 1-42949673055 2-64424509532 3-489626271746 4-107374182472 5-485331304451 6-150323855420 7-171798691894 8-193273528367 9-459561500679' 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836583 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836583 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836583 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836583' 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: waiting osd.0 seq 21474836583 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836584 -lt 21474836583 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673055 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673055 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:43.870 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673055 2024-03-09T13:31:43.871 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-03-09T13:31:43.871 INFO:tasks.workunit.client.0.smithi162.stderr:236466: waiting osd.1 seq 42949673055 2024-03-09T13:31:43.871 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:31:43.871 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673056 -lt 42949673055 2024-03-09T13:31:43.871 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:43.871 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509532 2024-03-09T13:31:43.871 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:43.871 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:31:44.762 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509532 2024-03-09T13:31:44.762 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:44.762 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509532 2024-03-09T13:31:44.762 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-03-09T13:31:44.762 INFO:tasks.workunit.client.0.smithi162.stderr:236466: waiting osd.2 seq 64424509532 2024-03-09T13:31:44.762 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:31:44.762 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509532 -lt 64424509532 2024-03-09T13:31:44.762 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:44.762 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-489626271746 2024-03-09T13:31:44.762 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-489626271746 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271746 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 489626271746' 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: waiting osd.3 seq 489626271746 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271747 -lt 489626271746 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182472 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182472 2024-03-09T13:31:44.763 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:44.764 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182472 2024-03-09T13:31:44.764 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182472' 2024-03-09T13:31:44.764 INFO:tasks.workunit.client.0.smithi162.stderr:236466: waiting osd.4 seq 107374182472 2024-03-09T13:31:44.764 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:31:44.764 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182473 -lt 107374182472 2024-03-09T13:31:44.764 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:44.764 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-485331304451 2024-03-09T13:31:44.764 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:44.764 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:31:44.764 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-485331304451 2024-03-09T13:31:44.764 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:44.764 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304451 2024-03-09T13:31:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509531 2024-03-09T13:31:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509531 2024-03-09T13:31:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-03-09T13:31:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:236464: waiting osd.2 seq 64424509531 2024-03-09T13:31:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:31:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509532 -lt 64424509531 2024-03-09T13:31:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-489626271747 2024-03-09T13:31:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-489626271747 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271747 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 489626271747' 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: waiting osd.3 seq 489626271747 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271747 -lt 489626271747 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182473 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182473 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182473 2024-03-09T13:31:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182473' 2024-03-09T13:31:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:236464: waiting osd.4 seq 107374182473 2024-03-09T13:31:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:31:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182473 -lt 107374182473 2024-03-09T13:31:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-485331304450 2024-03-09T13:31:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:31:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-485331304450 2024-03-09T13:31:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=485331304450 2024-03-09T13:31:46.018 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'wai 'waiting osd.5 seq 485331304451' 2024-03-09T13:31:46.018 INFO:tasks.workunit.client.0.smithi162.stderr:236466: waiting osd.5 seq 485331304451 2024-03-09T13:31:46.018 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:31:46.018 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304451 -lt 485331304451 2024-03-09T13:31:46.018 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:46.018 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855420 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855420 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855420 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855420' 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: waiting osd.6 seq 150323855420 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855422 -lt 150323855420 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691894 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:31:46.019 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691894 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691894 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: waiting osd.7 seq 171798691894 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691895 -lt 171798691894 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528367 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528367 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528367 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528367' 2024-03-09T13:31:46.020 INFO:tasks.workunit.client.0.smithi162.stderr:236466: waiting osd.8 seq 193273528367 2024-03-09T13:31:46.021 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:31:46.021 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528368 -lt 193273528367 2024-03-09T13:31:46.100 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: fting osd.5 seq 485331304450' 2024-03-09T13:31:46.100 INFO:tasks.workunit.client.0.smithi162.stderr:236464: waiting osd.5 seq 485331304450 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 485331304451 -lt 485331304450 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855421 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855421 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855421 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855421' 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: waiting osd.6 seq 150323855421 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855422 -lt 150323855421 2024-03-09T13:31:46.101 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691895 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691895 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691895 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691895' 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: waiting osd.7 seq 171798691895 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691895 -lt 171798691895 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528368 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:46.102 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:31:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528368 2024-03-09T13:31:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528368 2024-03-09T13:31:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528368' 2024-03-09T13:31:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:236464: waiting osd.8 seq 193273528368 2024-03-09T13:31:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:31:46.103 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528368 -lt 193273528368 2024-03-09T13:31:47.346 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_lush_pg_stats: for s in $seqs 2024-03-09T13:31:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500679 2024-03-09T13:31:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:31:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500679 2024-03-09T13:31:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500679 2024-03-09T13:31:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500679' 2024-03-09T13:31:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:236466: waiting osd.9 seq 459561500679 2024-03-09T13:31:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:31:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500680 -lt 459561500679 2024-03-09T13:31:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:31:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:31:47.348 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:31:47.349 INFO:tasks.workunit.client.0.smithi162.stderr:236466: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:31:47.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:31:47.390 INFO:tasks.workunit.client.0.smithi162.stderr:pg_stats: for s in $seqs 2024-03-09T13:31:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500680 2024-03-09T13:31:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:31:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:31:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500680 2024-03-09T13:31:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:31:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500680 2024-03-09T13:31:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500680' 2024-03-09T13:31:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:236464: waiting osd.9 seq 459561500680 2024-03-09T13:31:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:31:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500680 -lt 459561500680 2024-03-09T13:31:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:31:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:47.391 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:31:47.392 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr:236464: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 236463 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:31:47.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:31:47.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T13:31:47.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T13:31:47.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:31:47.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:31:47.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:31:47.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:31:47.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:31:47.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:31:47.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:31:15.371099+0000 2024-03-09T13:31:47.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:31:48.206 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:31:48.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:31:15.371099+0000 2024-03-09T13:31:48.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:31:48.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:31:15.371099+0000 2024-03-09T13:31:48.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:31:48.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:31:48.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:31:48.218 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:31:48.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:31:48.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:31:48.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:31:48.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:31:48.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:31:15.371099+0000 '>' 2024-03-09T13:31:15.371099+0000 2024-03-09T13:31:48.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:31:49.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:31:49.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:31:49.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:31:49.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:31:49.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:31:49.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:31:49.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:31:49.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:31:15.371099+0000 '>' 2024-03-09T13:31:15.371099+0000 2024-03-09T13:31:49.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:31:50.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:31:50.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:31:50.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:31:50.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:31:50.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:31:50.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:31:50.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:31:51.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:31:48.305596+0000 '>' 2024-03-09T13:31:15.371099+0000 2024-03-09T13:31:51.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:31:51.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T13:31:51.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:31:51.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:31:51.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 239817"' 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 239817' 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 239818"' 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 239818' 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 239817 239818' 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 239817 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/239820: /' 2024-03-09T13:31:51.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:31:51.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/239822: /' 2024-03-09T13:31:53.879 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:31:53.880 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:31:53.880 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:31:53.880 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:31:53.880 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:31:53.880 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:31:53.880 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:31:53.880 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:31:53.880 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:31:53.880 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:31:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:31:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:31:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:31:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:31:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:31:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:31:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:31:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:31:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:31:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:31:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:31:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:31:53.882 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T13:31:53.882 INFO:tasks.workunit.client.0.smithi162.stderr:239820: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T13:31:53.882 INFO:tasks.workunit.client.0.smithi162.stderr:239820: _ 2024-03-09T13:31:53.882 INFO:tasks.workunit.client.0.smithi162.stderr:239820: hinfo_key 2024-03-09T13:31:53.882 INFO:tasks.workunit.client.0.smithi162.stderr:239820: snapset 2024-03-09T13:31:53.882 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:31:53.882 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:31:53.882 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:31:53.882 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:31:53.882 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:31:53.883 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:31:53.883 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:31:53.883 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:31:53.883 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:31:53.883 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:31:53.883 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:31:53.883 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:31:53.884 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:31:53.884 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:31:53.884 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:31:53.884 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:31:53.884 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:31:53.884 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:31:53.884 INFO:tasks.workunit.client.0.smithi162.stderr:239820: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:31:53.884 INFO:tasks.workunit.client.0.smithi162.stderr:239820: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:31:53.884 INFO:tasks.workunit.client.0.smithi162.stderr:239820: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:31:53.884 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:53.884 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:53.885 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:31:53.885 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:31:53.885 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:31:53.885 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:31:53.885 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:31:53.885 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:31:53.885 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:31:53.885 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:31:53.886 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:31:53.886 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:31:53.886 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:31:53.886 INFO:tasks.workunit.client.0.smithi162.stderr:239820: start osd.3 2024-03-09T13:31:54.275 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:31:54.276 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:31:54.276 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T13:31:54.276 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:31:54.276 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:31:54.276 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:31:54.276 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:31:54.276 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T13:31:54.276 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:31:54.276 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T13:31:54.276 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:31:54.276 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:31:54.276 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: _ 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: hinfo_key 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: snapset 2024-03-09T13:31:54.277 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T13:31:54.278 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:31:54.278 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:31:54.278 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T13:31:54.278 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:31:54.278 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:31:54.278 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:31:54.278 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:31:54.278 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:31:54.278 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:31:54.278 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:31:54.279 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:31:54.280 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:31:54.280 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:31:54.280 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:31:54.280 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:31:54.280 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:31:54.280 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:31:54.280 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:31:54.280 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T13:31:54.280 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T13:31:54.280 INFO:tasks.workunit.client.0.smithi162.stderr:239822: start osd.5 2024-03-09T13:31:58.097 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:31:58.097 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:31:58.097 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:31:58.097 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:31:58.097 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:31:58.097 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:31:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 2024-03-09T13:31:53.910+0000 7fe913a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 2024-03-09T13:31:53.924+0000 7fe913a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 2024-03-09T13:31:53.926+0000 7fe913a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:31:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:31:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:31:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:31:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:31:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:31:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 0 2024-03-09T13:31:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:58.098 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 2024-03-09T13:31:54.440+0000 7fe913a15640 -1 Falling back to public interface 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 1 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 2 2024-03-09T13:31:58.099 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:58.100 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:31:58.100 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 2024-03-09T13:31:56.828+0000 7fe913a15640 -1 osd.3 115 log_to_monitors true 2024-03-09T13:31:58.100 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:58.100 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:58.100 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:58.100 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:31:58.100 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 3 2024-03-09T13:31:58.100 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:58.961 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-03-09T13:31:58.961 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T13:31:58.961 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T13:31:58.961 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:31:58.961 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:31:58.961 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:31:58.961 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 2024-03-09T13:31:54.303+0000 7f4379aaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 2024-03-09T13:31:54.305+0000 7f4379aaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 2024-03-09T13:31:54.306+0000 7f4379aaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 0 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 2024-03-09T13:31:54.827+0000 7f4379aaa640 -1 Falling back to public interface 2024-03-09T13:31:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 1 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 2 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:31:58.964 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 2024-03-09T13:31:57.709+0000 7f4379aaa640 -1 osd.5 115 log_to_monitors true 2024-03-09T13:31:58.964 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:31:58.964 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:31:58.964 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:31:58.964 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 3 2024-03-09T13:31:58.964 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:31:58.964 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:31:58.980 INFO:tasks.workunit.client.0.smithi162.stderr:239822: osd.5 up in weight 1 up_from 121 up_thru 113 dowrs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:31:58.980 INFO:tasks.workunit.client.0.smithi162.stderr:239820: osd.3 up in weight 1 up_from 119 up_thru 119 down_at 116 last_clean_interval [114,115) [v2:127.0.0.1:6826/3009983891,v1:127.0.0.1:6827/3009983891] [v2:127.0.0.1:6828/3009983891,v1:127.0.0.1:6829/3009983891] exists,up e89c8cb7-e250-4035-b64e-470d42b71834 2024-03-09T13:31:58.980 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:31:58.980 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:31:58.980 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:31:58.980 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:31:58.981 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 1 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 2 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 3 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 4 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 5 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 6 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 7 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 8 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: 9' 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:58.982 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:31:58.983 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836588 2024-03-09T13:31:58.983 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836588 2024-03-09T13:31:58.983 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588' 2024-03-09T13:31:58.983 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:58.983 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:31:58.983 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673060 2024-03-09T13:31:58.983 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673060 2024-03-09T13:31:58.983 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673060' 2024-03-09T13:31:59.909 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0n_at 116 last_clean_interval [113,115) [v2:127.0.0.1:6874/673562901,v1:127.0.0.1:6875/673562901] [v2:127.0.0.1:6876/673562901,v1:127.0.0.1:6877/673562901] exists,up 9fc21c30-bd92-43ad-a6a2-73fee02bea12 2024-03-09T13:31:59.909 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:31:59.909 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:31:59.909 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:31:59.909 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:31:59.909 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:31:59.909 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:31:59.909 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:31:59.909 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:31:59.909 INFO:tasks.workunit.client.0.smithi162.stderr:239822: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:31:59.909 INFO:tasks.workunit.client.0.smithi162.stderr:239822: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:31:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:31:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:31:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:31:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:31:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:31:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:31:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:31:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:31:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:31:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:31:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 1 2024-03-09T13:31:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 2 2024-03-09T13:31:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 3 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 4 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 5 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 6 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 7 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 8 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: 9' 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836589 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836589 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589' 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:31:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:31:59.912 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673061 2024-03-09T13:31:59.912 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673061 2024-03-09T13:31:59.912 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673061' 2024-03-09T13:31:59.912 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:00.347 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clon/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:00.347 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:32:00.347 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509536 2024-03-09T13:32:00.347 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509536 2024-03-09T13:32:00.347 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673060 2-64424509536' 2024-03-09T13:32:00.347 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:00.347 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=511101108226 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 511101108226 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673060 2-64424509536 3-511101108226' 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182478 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182478 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673060 2-64424509536 3-511101108226 4-107374182478' 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042818 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042818 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673060 2-64424509536 3-511101108226 4-107374182478 5-519691042818' 2024-03-09T13:32:00.348 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:00.349 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:32:00.349 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855426 2024-03-09T13:32:00.349 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855426 2024-03-09T13:32:00.349 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673060 2-64424509536 3-511101108226 4-107374182478 5-519691042818 6-150323855426' 2024-03-09T13:32:00.349 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:00.349 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:32:00.349 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691899 2024-03-09T13:32:00.349 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691899 2024-03-09T13:32:00.349 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673060 2-64424509536 3-511101108226 4-107374182478 5-519691042818 6-150323855426 7-171798691899' 2024-03-09T13:32:00.349 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:01.046 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qe.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:32:01.047 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509537 2024-03-09T13:32:01.047 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509537 2024-03-09T13:32:01.047 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673061 2-64424509537' 2024-03-09T13:32:01.047 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:01.047 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:32:01.047 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=511101108227 2024-03-09T13:32:01.047 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 511101108227 2024-03-09T13:32:01.047 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673061 2-64424509537 3-511101108227' 2024-03-09T13:32:01.047 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:01.047 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:32:01.047 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182479 2024-03-09T13:32:01.047 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182479 2024-03-09T13:32:01.047 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673061 2-64424509537 3-511101108227 4-107374182479' 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042819 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042819 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673061 2-64424509537 3-511101108227 4-107374182479 5-519691042819' 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855427 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855427 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673061 2-64424509537 3-511101108227 4-107374182479 5-519691042819 6-150323855427' 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691900 2024-03-09T13:32:01.048 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691900 2024-03-09T13:32:01.049 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673061 2-64424509537 3-511101108227 4-107374182479 5-519691042819 6-150323855427 7-171798691900' 2024-03-09T13:32:01.049 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:01.049 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:32:01.297 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/ca/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:32:01.297 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528372 2024-03-09T13:32:01.297 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528372 2024-03-09T13:32:01.297 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673060 2-64424509536 3-511101108226 4-107374182478 5-519691042818 6-150323855426 7-171798691899 8-193273528372' 2024-03-09T13:32:01.297 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:01.297 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:32:01.298 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500685 2024-03-09T13:32:01.298 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500685 2024-03-09T13:32:01.298 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673060 2-64424509536 3-511101108226 4-107374182478 5-519691042818 6-150323855426 7-171798691899 8-193273528372 9-459561500685' 2024-03-09T13:32:01.298 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:01.298 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836588 2024-03-09T13:32:01.298 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:01.298 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:32:01.298 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836588 2024-03-09T13:32:01.298 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:01.298 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836588 2024-03-09T13:32:01.298 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836588' 2024-03-09T13:32:01.298 INFO:tasks.workunit.client.0.smithi162.stderr:239820: waiting osd.0 seq 21474836588 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836590 -lt 21474836588 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673060 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673060 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673060 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673060' 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: waiting osd.1 seq 42949673060 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673061 -lt 42949673060 2024-03-09T13:32:01.299 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:01.300 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509536 2024-03-09T13:32:01.300 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:01.772 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cepephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528374 2024-03-09T13:32:01.772 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528374 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673061 2-64424509537 3-511101108227 4-107374182479 5-519691042819 6-150323855427 7-171798691900 8-193273528374' 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500686 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500686 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673061 2-64424509537 3-511101108227 4-107374182479 5-519691042819 6-150323855427 7-171798691900 8-193273528374 9-459561500686' 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836589 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836589 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836589 2024-03-09T13:32:01.773 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: waiting osd.0 seq 21474836589 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836590 -lt 21474836589 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673061 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673061 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673061 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: waiting osd.1 seq 42949673061 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:32:01.774 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673061 -lt 42949673061 2024-03-09T13:32:01.775 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:01.775 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509537 2024-03-09T13:32:01.775 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:01.775 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:32:02.221 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:32:02.221 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509536 2024-03-09T13:32:02.221 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:02.221 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509536 2024-03-09T13:32:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-03-09T13:32:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:239820: waiting osd.2 seq 64424509536 2024-03-09T13:32:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:32:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509536 2024-03-09T13:32:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-511101108226 2024-03-09T13:32:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:32:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-511101108226 2024-03-09T13:32:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=511101108226 2024-03-09T13:32:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 511101108226' 2024-03-09T13:32:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:239820: waiting osd.3 seq 511101108226 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 511101108227 -lt 511101108226 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182478 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182478 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182478 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182478' 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: waiting osd.4 seq 107374182478 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182479 -lt 107374182478 2024-03-09T13:32:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:02.224 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-519691042818 2024-03-09T13:32:02.224 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:02.224 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:32:02.224 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-519691042818 2024-03-09T13:32:02.224 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:02.718 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=5196htest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509537 2024-03-09T13:32:02.718 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:02.718 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509537 2024-03-09T13:32:02.718 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509537' 2024-03-09T13:32:02.718 INFO:tasks.workunit.client.0.smithi162.stderr:239822: waiting osd.2 seq 64424509537 2024-03-09T13:32:02.718 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:32:02.718 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509537 -lt 64424509537 2024-03-09T13:32:02.718 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-511101108227 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-511101108227 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=511101108227 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 511101108227' 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: waiting osd.3 seq 511101108227 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 511101108227 -lt 511101108227 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182479 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:02.719 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182479 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182479 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: waiting osd.4 seq 107374182479 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182479 -lt 107374182479 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-519691042819 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-519691042819 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=519691042819 2024-03-09T13:32:03.502 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats:91042818 2024-03-09T13:32:03.502 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 519691042818' 2024-03-09T13:32:03.502 INFO:tasks.workunit.client.0.smithi162.stderr:239820: waiting osd.5 seq 519691042818 2024-03-09T13:32:03.502 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:32:03.502 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042819 -lt 519691042818 2024-03-09T13:32:03.502 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:03.502 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855426 2024-03-09T13:32:03.502 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:03.502 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855426 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855426 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855426' 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: waiting osd.6 seq 150323855426 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855426 -lt 150323855426 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691899 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691899 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:03.503 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691899 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691899' 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: waiting osd.7 seq 171798691899 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691900 -lt 171798691899 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528372 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528372 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528372 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528372' 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: waiting osd.8 seq 193273528372 2024-03-09T13:32:03.504 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:32:03.954 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 echo 'waiting osd.5 seq 519691042819' 2024-03-09T13:32:03.955 INFO:tasks.workunit.client.0.smithi162.stderr:239822: waiting osd.5 seq 519691042819 2024-03-09T13:32:03.955 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:32:03.955 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042819 -lt 519691042819 2024-03-09T13:32:03.955 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:03.955 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855427 2024-03-09T13:32:03.955 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:03.955 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:32:03.955 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855427 2024-03-09T13:32:03.955 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:03.955 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855427 2024-03-09T13:32:03.955 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855427' 2024-03-09T13:32:03.955 INFO:tasks.workunit.client.0.smithi162.stderr:239822: waiting osd.6 seq 150323855427 2024-03-09T13:32:03.955 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:32:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855427 -lt 150323855427 2024-03-09T13:32:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691900 2024-03-09T13:32:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:32:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691900 2024-03-09T13:32:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691900 2024-03-09T13:32:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-03-09T13:32:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:239822: waiting osd.7 seq 171798691900 2024-03-09T13:32:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:32:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691900 -lt 171798691900 2024-03-09T13:32:03.956 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528374 2024-03-09T13:32:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:32:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528374 2024-03-09T13:32:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528374 2024-03-09T13:32:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-03-09T13:32:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:239822: waiting osd.8 seq 193273528374 2024-03-09T13:32:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:32:03.957 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528374 -lt 193273528374 2024-03-09T13:32:04.865 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:293273528374 -lt 193273528372 2024-03-09T13:32:04.865 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:04.865 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500685 2024-03-09T13:32:04.865 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:04.865 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:32:04.865 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500685 2024-03-09T13:32:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500685 2024-03-09T13:32:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500685' 2024-03-09T13:32:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:239820: waiting osd.9 seq 459561500685 2024-03-09T13:32:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:32:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500686 -lt 459561500685 2024-03-09T13:32:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:32:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:32:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:32:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:32:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:32:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:32:04.866 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:239820: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:239820: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:32:04.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 239818 2024-03-09T13:32:05.257 INFO:tasks.workunit.client.0.smithi162.stderr:210: flush_pg_stats: for s in $seqs 2024-03-09T13:32:05.257 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500686 2024-03-09T13:32:05.257 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:32:05.257 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:32:05.257 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500686 2024-03-09T13:32:05.257 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:32:05.257 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500686 2024-03-09T13:32:05.257 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500686' 2024-03-09T13:32:05.257 INFO:tasks.workunit.client.0.smithi162.stderr:239822: waiting osd.9 seq 459561500686 2024-03-09T13:32:05.257 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:32:05.257 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500686 -lt 459561500686 2024-03-09T13:32:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:32:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:32:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:32:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:32:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:32:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:32:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:32:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:32:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:32:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:32:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:32:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:32:05.258 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:32:05.259 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:32:05.259 INFO:tasks.workunit.client.0.smithi162.stderr:239822: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:32:05.259 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:32:05.259 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:32:05.259 INFO:tasks.workunit.client.0.smithi162.stderr:239822: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:32:05.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:32:05.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:32:05.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:32:05.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:32:05.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T13:32:05.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:32:05.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:32:05.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:32:05.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:32:05.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:32:05.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:32:05.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:32:05.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:32:05.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:32:05.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:32:05.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:32:05.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:32:05.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:32:05.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:32:05.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:32:05.425 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:32:05.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:32:05.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:32:05.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:32:05.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:32:05.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:32:05.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:32:05.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:32:05.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:32:05.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:32:05.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:32:05.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:32:05.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:32:05.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:32:05.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:32:05.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:32:05.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:32:05.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:32:05.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:32:05.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:32:05.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:32:05.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:32:05.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:32:05.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:32:05.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:32:05.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:32:05.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:32:05.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:32:05.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:32:05.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:32:05.492 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:32:05.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:32:05.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:32:05.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:32:05.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:32:05.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:32:05.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:32:05.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:32:05.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:32:05.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:32:05.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:32:05.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:32:05.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:32:05.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:32:05.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:32:05.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:32:05.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:32:05.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:32:05.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:32:05.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:32:05.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:32:05.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:32:05.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:32:05.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:32:05.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:32:05.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:32:05.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:32:05.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:32:05.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2024-03-09T13:32:05.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2024-03-09T13:32:05.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2024-03-09T13:32:05.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2024-03-09T13:32:05.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: corrupt_and_repair_lrc: local allow_overwrites=true 2024-03-09T13:32:05.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: corrupt_and_repair_lrc: local poolname=ecpool 2024-03-09T13:32:05.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2024-03-09T13:32:05.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:32:05.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:32:05.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:32:05.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:32:05.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:32:05.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T13:32:05.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:32:05.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:32:05.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:32:05.977 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:32:05.977 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:32:05.977 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:32:05.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:32:05.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T13:32:06.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:32:06.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:32:06.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:32:06.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:32:06.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:32:06.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:32:06.152 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:32:06.152 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:32:06.152 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:32:06.152 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:32:06.152 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:32:06.152 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:32:06.152 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:32:06.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:32:06.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:32:06.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:32:06.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:32:06.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:32:06.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:32:06.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:32:06.232 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:32:06.232 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:32:06.232 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:32:06.232 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:32:06.232 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:32:06.232 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:32:06.232 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:32:06.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:32:06.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:32:06.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2024-03-09T13:32:06.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:32:06.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:32:06.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:32:06.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:32:06.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:32:06.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:32:07.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:32:07.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:32:07.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:32:07.119 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:32:07.119 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:32:07.119 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:32:07.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:32:07.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:32:07.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:32:07.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: seq 0 9 2024-03-09T13:32:07.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:32:07.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2024-03-09T13:32:07.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:32:07.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:32:07.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:32:07.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:32:07.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:32:07.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:32:07.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:32:07.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:32:07.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:32:07.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:32:07.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:32:07.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:32:07.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:32:07.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:32:07.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:32:07.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:32:07.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:32:07.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:32:07.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:32:07.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:32:07.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:32:07.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=56063998-9d47-430f-8b13-ff389f83a475 2024-03-09T13:32:07.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 56063998-9d47-430f-8b13-ff389f83a475' 2024-03-09T13:32:07.154 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 56063998-9d47-430f-8b13-ff389f83a475 2024-03-09T13:32:07.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:32:07.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXZOxlJnASChAAtvj3gIOkk8GM2NGoo3VCkw== 2024-03-09T13:32:07.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXZOxlJnASChAAtvj3gIOkk8GM2NGoo3VCkw=="}' 2024-03-09T13:32:07.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 56063998-9d47-430f-8b13-ff389f83a475 -i td/osd-scrub-repair/0/new.json 2024-03-09T13:32:07.417 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:32:07.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:32:07.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXZOxlJnASChAAtvj3gIOkk8GM2NGoo3VCkw== --osd-uuid 56063998-9d47-430f-8b13-ff389f83a475 2024-03-09T13:32:07.455 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:07.455+0000 7f8a110c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:32:07.467 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:07.467+0000 7f8a110c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:32:07.468 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:07.468+0000 7f8a110c8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:32:07.468 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:07.469+0000 7f8a110c8640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:32:07.469 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:07.469+0000 7f8a110c8640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:32:07.469 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:07.469+0000 7f8a110c8640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:32:07.559 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:07.559+0000 7f8a110c8640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:32:13.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:32:13.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:32:13.124 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:32:13.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:32:13.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:32:13.485 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:32:13.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:32:13.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:32:13.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:32:13.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:32:13.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:32:13.511 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:13.511+0000 7f0861393640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:32:13.513 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:13.513+0000 7f0861393640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:32:13.514 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:13.514+0000 7f0861393640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:32:13.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:13.759 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:32:13.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:32:13.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:13.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:14.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:14.045 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:14.046+0000 7f0861393640 -1 Falling back to public interface 2024-03-09T13:32:15.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:15.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:15.031 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:32:15.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:32:15.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:15.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:15.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:16.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:16.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:16.306 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:32:16.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:32:16.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:16.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:16.373 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:16.373+0000 7f0861393640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:32:16.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:17.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:17.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:17.642 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:32:17.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:32:17.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:17.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:17.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:18.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:18.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:18.926 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:32:18.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:32:18.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:18.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:19.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:20.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:20.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:20.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:32:20.324 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:32:20.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:20.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:20.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:21.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:21.608 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:32:21.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:21.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:32:21.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:21.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:21.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:22.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:22.955 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:32:22.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:22.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:32:22.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:22.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:23.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:24.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:24.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:24.299 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:32:24.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:32:24.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:24.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:24.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:25.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:25.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:25.580 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:32:25.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:32:25.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:25.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:25.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:26.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:26.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:26.930 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:32:26.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:32:26.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:26.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:27.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:28.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:28.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:28.319 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:32:28.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:32:28.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:28.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:28.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:29.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:29.722 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:32:29.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:29.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:32:29.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:29.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:30.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:31.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:31.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:31.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:32:31.285 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:32:31.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:31.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:32.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:33.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:33.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:33.153 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:32:33.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:32:33.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:33.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:33.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:34.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:34.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:34.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:32:34.606 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:32:34.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:34.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:35.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:36.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:36.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:36.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:32:36.034 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:32:36.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:36.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:36.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:37.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:37.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:37.487 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:32:37.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:32:37.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:37.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:38.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:39.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:39.338 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:32:39.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:39.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:32:39.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:39.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:32:39.622 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3892548510,v1:127.0.0.1:6803/3892548510] [v2:127.0.0.1:6804/3892548510,v1:127.0.0.1:6805/3892548510] exists,up 56063998-9d47-430f-8b13-ff389f83a475 2024-03-09T13:32:39.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:32:39.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:32:39.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:32:39.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:32:39.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2024-03-09T13:32:39.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:32:39.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:32:39.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:32:39.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:32:39.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:32:39.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:32:39.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:32:39.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:32:39.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:32:39.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:32:39.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:32:39.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:32:39.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:32:39.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:32:39.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:32:39.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:32:39.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:32:39.628 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:32:39.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:32:39.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:32:39.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:32:39.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:32:39.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=745a9bfd-8b46-4d38-bea0-51b289566a62 2024-03-09T13:32:39.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 745a9bfd-8b46-4d38-bea0-51b289566a62' 2024-03-09T13:32:39.630 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 745a9bfd-8b46-4d38-bea0-51b289566a62 2024-03-09T13:32:39.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:32:39.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3ZOxl77dTJhAAhWwGNa1gBJur0Fqcv7oxzg== 2024-03-09T13:32:39.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3ZOxl77dTJhAAhWwGNa1gBJur0Fqcv7oxzg=="}' 2024-03-09T13:32:39.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 745a9bfd-8b46-4d38-bea0-51b289566a62 -i td/osd-scrub-repair/1/new.json 2024-03-09T13:32:40.014 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:32:40.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:32:40.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3ZOxl77dTJhAAhWwGNa1gBJur0Fqcv7oxzg== --osd-uuid 745a9bfd-8b46-4d38-bea0-51b289566a62 2024-03-09T13:32:40.054 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:40.054+0000 7fe2492d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:32:40.099 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:40.098+0000 7fe2492d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:32:40.100 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:40.099+0000 7fe2492d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:32:40.100 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:40.099+0000 7fe2492d9640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:32:40.100 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:40.099+0000 7fe2492d9640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:32:40.101 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:40.099+0000 7fe2492d9640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:32:40.166 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:40.165+0000 7fe2492d9640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:32:45.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:32:45.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:32:45.981 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:32:45.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:32:45.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:32:46.334 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:32:46.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:32:46.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:32:46.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:32:46.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:32:46.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:32:46.360 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:46.360+0000 7f4179973640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:32:46.362 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:46.362+0000 7f4179973640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:32:46.363 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:46.363+0000 7f4179973640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:32:46.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:32:46.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:32:46.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:32:46.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:32:46.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:32:46.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:46.610 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:32:46.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:32:46.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:46.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:32:46.879 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:46.880+0000 7f4179973640 -1 Falling back to public interface 2024-03-09T13:32:46.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:47.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:47.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:47.883 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:32:47.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:32:47.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:47.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:32:48.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:49.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:49.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:49.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:32:49.155 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:32:49.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:49.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:32:49.221 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:32:49.221+0000 7f4179973640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:32:49.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:50.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:50.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:50.439 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:32:50.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:32:50.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:50.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:32:50.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:51.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:51.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:51.725 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:32:51.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:32:51.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:51.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:32:52.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:53.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:53.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:53.013 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:32:53.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:32:53.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:53.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:32:53.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:54.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:54.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:54.326 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:32:54.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:32:54.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:54.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:32:54.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:55.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:55.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:55.610 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:32:55.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:32:55.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:55.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:32:55.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:56.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:32:56.951 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:32:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:56.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:32:57.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:58.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:58.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:58.353 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:32:58.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:32:58.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:58.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:32:58.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:32:59.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:32:59.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:32:59.636 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:32:59.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:32:59.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:32:59.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:32:59.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:00.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:00.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:00.977 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:33:00.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:33:00.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:00.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:33:01.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:02.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:02.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:02.362 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:33:02.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:33:02.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:02.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:33:02.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:03.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:03.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:03.758 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:33:03.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:33:03.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:03.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:33:04.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:05.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:05.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:05.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:33:05.446 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:33:05.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:05.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:33:06.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:07.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:07.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:07.223 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:33:07.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:33:07.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:07.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:33:07.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:08.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:08.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:08.658 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:33:08.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:33:08.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:08.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:33:09.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:10.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:10.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:10.328 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:33:10.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:33:10.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:10.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:33:10.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:11.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:11.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:11.874 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:33:11.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:33:11.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:11.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:33:12.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:13.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:13.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:13.426 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:33:13.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:33:13.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:13.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:33:13.707 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/762572070,v1:127.0.0.1:6811/762572070] [v2:127.0.0.1:6812/762572070,v1:127.0.0.1:6813/762572070] exists,up 745a9bfd-8b46-4d38-bea0-51b289566a62 2024-03-09T13:33:13.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:33:13.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:33:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:33:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:33:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2024-03-09T13:33:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:33:13.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:33:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T13:33:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:33:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:33:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:33:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:33:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:33:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:33:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:33:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:33:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:33:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:33:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:33:13.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:33:13.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:33:13.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:33:13.714 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:33:13.714 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:33:13.714 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:33:13.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:33:13.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:33:13.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:33:13.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:33:13.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:33:13.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:33:13.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:33:13.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:33:13.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:33:13.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:33:13.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:33:13.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:33:13.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:33:13.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:33:13.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:33:13.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=25d6cf13-358c-4fd4-8d95-e344080536fd 2024-03-09T13:33:13.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 25d6cf13-358c-4fd4-8d95-e344080536fd' 2024-03-09T13:33:13.716 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 25d6cf13-358c-4fd4-8d95-e344080536fd 2024-03-09T13:33:13.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:33:13.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZZexlqL9pKxAAt9IFnZ+4Pn53RMzfIVMnjg== 2024-03-09T13:33:13.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZZexlqL9pKxAAt9IFnZ+4Pn53RMzfIVMnjg=="}' 2024-03-09T13:33:13.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 25d6cf13-358c-4fd4-8d95-e344080536fd -i td/osd-scrub-repair/2/new.json 2024-03-09T13:33:14.085 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:33:14.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T13:33:14.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZZexlqL9pKxAAt9IFnZ+4Pn53RMzfIVMnjg== --osd-uuid 25d6cf13-358c-4fd4-8d95-e344080536fd 2024-03-09T13:33:14.125 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:14.124+0000 7fe558793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:33:14.170 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:14.169+0000 7fe558793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:33:14.171 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:14.170+0000 7fe558793640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:33:14.171 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:14.171+0000 7fe558793640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:33:14.172 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:14.171+0000 7fe558793640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:33:14.172 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:14.171+0000 7fe558793640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:33:14.245 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:14.244+0000 7fe558793640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T13:33:20.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T13:33:20.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:33:20.301 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T13:33:20.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T13:33:20.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:33:20.655 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T13:33:20.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T13:33:20.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:33:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:33:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:33:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:33:20.681 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:20.680+0000 7f534b81f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:33:20.683 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:20.682+0000 7f534b81f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:33:20.684 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:20.683+0000 7f534b81f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:33:20.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T13:33:20.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:33:20.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:33:20.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:33:20.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:33:20.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:20.928 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:33:20.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:33:20.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:20.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:21.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:21.204 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:21.204+0000 7f534b81f640 -1 Falling back to public interface 2024-03-09T13:33:22.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:22.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:22.205 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:33:22.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:33:22.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:22.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:22.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:23.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:23.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:23.491 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:33:23.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:33:23.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:23.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:23.695 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:23.694+0000 7f534b81f640 -1 osd.2 0 log_to_monitors true 2024-03-09T13:33:23.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:24.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:24.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:24.788 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:33:24.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:33:24.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:24.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:25.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:26.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:26.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:26.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:33:26.071 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:33:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:26.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:26.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:27.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:27.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:27.360 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:33:27.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:33:27.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:27.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:27.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:28.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:28.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:28.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:33:28.637 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:33:28.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:28.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:28.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:29.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:29.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:29.922 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:33:29.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:33:29.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:29.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:30.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:31.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:31.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:31.205 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:33:31.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:33:31.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:31.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:31.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:32.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:32.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:32.581 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:33:32.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:33:32.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:32.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:32.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:33.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:33.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:33.866 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:33:33.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:33:33.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:33.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:34.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:35.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:35.149 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:33:35.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:35.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:33:35.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:35.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:35.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:36.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:36.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:36.838 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:33:36.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:33:36.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:36.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:37.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:38.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:38.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:38.299 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:33:38.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:33:38.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:38.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:39.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:40.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:40.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:40.026 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:33:40.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:33:40.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:40.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:40.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:41.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:41.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:41.738 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:33:41.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:33:41.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:41.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:42.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:43.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:43.649 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:33:43.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:33:43.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:43.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:44.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:45.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:45.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:45.040 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:33:45.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:33:45.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:45.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:45.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:46.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:46.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:46.327 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:33:46.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:33:46.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:46.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:33:46.602 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3634055294,v1:127.0.0.1:6819/3634055294] [v2:127.0.0.1:6820/3634055294,v1:127.0.0.1:6821/3634055294] exists,up 25d6cf13-358c-4fd4-8d95-e344080536fd 2024-03-09T13:33:46.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:33:46.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:33:46.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:33:46.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:33:46.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2024-03-09T13:33:46.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:33:46.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:33:46.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:33:46.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:33:46.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:33:46.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:33:46.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:33:46.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:33:46.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2095e0ae-504c-4658-8991-61158f227389 2024-03-09T13:33:46.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 2095e0ae-504c-4658-8991-61158f227389' 2024-03-09T13:33:46.610 INFO:tasks.workunit.client.0.smithi162.stdout:add osd3 2095e0ae-504c-4658-8991-61158f227389 2024-03-09T13:33:46.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:33:46.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6ZexlVQ0xJRAApbxYnInoPceEOA08ItL/iQ== 2024-03-09T13:33:46.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6ZexlVQ0xJRAApbxYnInoPceEOA08ItL/iQ=="}' 2024-03-09T13:33:46.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2095e0ae-504c-4658-8991-61158f227389 -i td/osd-scrub-repair/3/new.json 2024-03-09T13:33:46.980 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:33:46.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/3/new.json 2024-03-09T13:33:46.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6ZexlVQ0xJRAApbxYnInoPceEOA08ItL/iQ== --osd-uuid 2095e0ae-504c-4658-8991-61158f227389 2024-03-09T13:33:47.020 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:47.020+0000 7f66e876b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:33:47.065 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:47.065+0000 7f66e876b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:33:47.066 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:47.066+0000 7f66e876b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:33:47.067 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:47.066+0000 7f66e876b640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T13:33:47.067 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:47.066+0000 7f66e876b640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T13:33:47.067 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:47.066+0000 7f66e876b640 -1 bluestore(td/osd-scrub-repair/3/block) _read_bdev_label failed to open td/osd-scrub-repair/3/block: (2) No such file or directory 2024-03-09T13:33:47.140 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:47.140+0000 7f66e876b640 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2024-03-09T13:33:52.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2024-03-09T13:33:53.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:33:53.092 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd3 key to auth repository 2024-03-09T13:33:53.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-03-09T13:33:53.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:33:53.235 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.3 2024-03-09T13:33:53.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-03-09T13:33:53.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:33:53.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:33:53.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:33:53.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:33:53.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:53.261+0000 7f5098e1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:33:53.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:53.274+0000 7f5098e1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:33:53.275 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:53.275+0000 7f5098e1f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:33:53.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-03-09T13:33:53.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:33:53.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:33:53.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:33:53.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:33:53.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:53.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:33:53.547 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:33:53.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:53.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:33:53.808 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:53.808+0000 7f5098e1f640 -1 Falling back to public interface 2024-03-09T13:33:53.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:54.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:54.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:54.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:33:54.854 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:33:54.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:54.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:33:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:56.126 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:33:56.125+0000 7f5098e1f640 -1 osd.3 0 log_to_monitors true 2024-03-09T13:33:56.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:56.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:56.137 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:33:56.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:33:56.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:56.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:33:56.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:57.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:57.452 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:33:57.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:57.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:33:57.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:57.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:33:57.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:33:58.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:33:58.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:33:58.850 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:33:58.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:33:58.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:33:58.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:33:59.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:00.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:00.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:00.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:34:00.210 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:34:00.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:00.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:00.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:01.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:01.496 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:34:01.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:01.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:34:01.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:01.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:01.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:02.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:02.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:02.797 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:34:02.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:34:02.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:02.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:03.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:04.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:04.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:04.118 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:34:04.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:34:04.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:04.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:04.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:05.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:05.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:05.407 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:34:05.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:34:05.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:05.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:05.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:06.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:06.691 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:34:06.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:06.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:34:06.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:06.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:06.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:07.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:07.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:07.977 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:34:07.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:34:07.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:07.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:08.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:09.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:09.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:09.263 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:34:09.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:34:09.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:09.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:10.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:11.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:11.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:11.135 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:34:11.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:34:11.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:11.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:11.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:12.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:12.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:12.963 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:34:12.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:34:12.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:12.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:13.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:14.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:14.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:14.249 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:34:14.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:34:14.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:14.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:14.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:15.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:15.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:15.841 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:34:15.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:34:15.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:15.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:16.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:17.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:17.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:17.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:34:17.456 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:34:17.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:17.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:18.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:19.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:19.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:19.299 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:34:19.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:34:19.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:19.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:34:19.596 INFO:tasks.workunit.client.0.smithi162.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2162713882,v1:127.0.0.1:6827/2162713882] [v2:127.0.0.1:6828/2162713882,v1:127.0.0.1:6829/2162713882] exists,up 2095e0ae-504c-4658-8991-61158f227389 2024-03-09T13:34:19.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:34:19.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:34:19.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:34:19.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:34:19.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2024-03-09T13:34:19.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:34:19.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:34:19.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-03-09T13:34:19.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:34:19.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/4 2024-03-09T13:34:19.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:34:19.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:34:19.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:34:19.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:34:19.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2024-03-09T13:34:19.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2024-03-09T13:34:19.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:34:19.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:34:19.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:34:19.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:34:19.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:34:19.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:34:19.602 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:34:19.602 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:34:19.602 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:34:19.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:34:19.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:34:19.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:34:19.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:34:19.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:34:19.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:34:19.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:34:19.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:34:19.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:34:19.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:34:19.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:34:19.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:34:19.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:34:19.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/4 2024-03-09T13:34:19.603 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:34:19.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0239c55a-742a-411c-b40f-a28838eeb523 2024-03-09T13:34:19.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 0239c55a-742a-411c-b40f-a28838eeb523' 2024-03-09T13:34:19.604 INFO:tasks.workunit.client.0.smithi162.stdout:add osd4 0239c55a-742a-411c-b40f-a28838eeb523 2024-03-09T13:34:19.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:34:19.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbZexltNXRJBAAG+pzyckRpj67O574hQT0AQ== 2024-03-09T13:34:19.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbZexltNXRJBAAG+pzyckRpj67O574hQT0AQ=="}' 2024-03-09T13:34:19.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0239c55a-742a-411c-b40f-a28838eeb523 -i td/osd-scrub-repair/4/new.json 2024-03-09T13:34:19.957 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:34:19.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/4/new.json 2024-03-09T13:34:19.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBbZexltNXRJBAAG+pzyckRpj67O574hQT0AQ== --osd-uuid 0239c55a-742a-411c-b40f-a28838eeb523 2024-03-09T13:34:19.997 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:19.997+0000 7f390256f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:34:20.034 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:20.034+0000 7f390256f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:34:20.034 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:20.035+0000 7f390256f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:34:20.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:20.035+0000 7f390256f640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-09T13:34:20.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:20.035+0000 7f390256f640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-09T13:34:20.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:20.035+0000 7f390256f640 -1 bluestore(td/osd-scrub-repair/4/block) _read_bdev_label failed to open td/osd-scrub-repair/4/block: (2) No such file or directory 2024-03-09T13:34:20.105 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:20.105+0000 7f390256f640 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2024-03-09T13:34:26.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2024-03-09T13:34:26.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:34:26.408 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd4 key to auth repository 2024-03-09T13:34:26.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-03-09T13:34:26.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:34:26.760 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.4 2024-03-09T13:34:26.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-03-09T13:34:26.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:34:26.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:34:26.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:34:26.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:34:26.787 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:26.787+0000 7f76ec0c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:34:26.788 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:26.789+0000 7f76ec0c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:34:26.789 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:26.790+0000 7f76ec0c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:34:27.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-03-09T13:34:27.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:34:27.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-03-09T13:34:27.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:34:27.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:34:27.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:27.040 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:34:27.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:34:27.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:27.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:27.304 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:27.304+0000 7f76ec0c2640 -1 Falling back to public interface 2024-03-09T13:34:27.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:28.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:28.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:28.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:34:28.358 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:34:28.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:28.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:28.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:29.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:29.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:29.641 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:34:29.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:34:29.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:29.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:29.795 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:29.795+0000 7f76ec0c2640 -1 osd.4 0 log_to_monitors true 2024-03-09T13:34:29.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:30.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:30.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:30.927 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:34:30.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:34:30.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:30.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:31.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:32.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:32.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:32.213 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:34:32.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:34:32.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:32.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:32.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:33.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:33.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:33.603 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:34:33.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:34:33.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:33.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:34.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:35.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:35.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:35.019 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:34:35.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:34:35.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:35.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:35.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:36.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:36.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:36.303 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:34:36.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:34:36.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:36.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:36.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:37.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:37.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:37.648 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:34:37.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:34:37.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:37.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:37.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:38.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:38.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:38.980 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:34:38.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:34:38.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:38.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:39.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:40.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:40.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:40.268 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:34:40.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:34:40.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:40.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:40.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:41.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:41.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:41.619 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:34:41.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:34:41.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:41.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:41.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:42.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:42.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:42.900 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:34:42.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:34:42.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:42.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:43.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:44.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:44.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:44.179 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:34:44.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:34:44.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:44.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:44.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:45.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:45.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:45.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:34:45.467 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:34:45.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:45.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:46.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:47.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:47.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:47.269 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:34:47.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:34:47.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:47.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:48.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:49.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:49.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:49.064 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:34:49.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:34:49.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:49.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:49.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:50.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:50.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:50.467 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:34:50.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:34:50.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:50.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:51.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:52.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:52.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:52.181 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:34:52.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:34:52.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:52.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:52.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:34:53.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:34:53.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:34:53.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:34:53.534 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:34:53.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:34:53.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-03-09T13:34:53.814 INFO:tasks.workunit.client.0.smithi162.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1325818473,v1:127.0.0.1:6835/1325818473] [v2:127.0.0.1:6836/1325818473,v1:127.0.0.1:6837/1325818473] exists,up 0239c55a-742a-411c-b40f-a28838eeb523 2024-03-09T13:34:53.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:34:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:34:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:34:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:34:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2024-03-09T13:34:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:34:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:34:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-03-09T13:34:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:34:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:34:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:34:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:34:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:34:53.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:34:53.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1a7a2a5c-3d7e-4a54-91c1-c81c5be3f535 2024-03-09T13:34:53.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 1a7a2a5c-3d7e-4a54-91c1-c81c5be3f535' 2024-03-09T13:34:53.822 INFO:tasks.workunit.client.0.smithi162.stdout:add osd5 1a7a2a5c-3d7e-4a54-91c1-c81c5be3f535 2024-03-09T13:34:53.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:34:53.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9Zexl5fDWMRAAmat701broixlHMIPUuYlJA== 2024-03-09T13:34:53.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9Zexl5fDWMRAAmat701broixlHMIPUuYlJA=="}' 2024-03-09T13:34:53.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1a7a2a5c-3d7e-4a54-91c1-c81c5be3f535 -i td/osd-scrub-repair/5/new.json 2024-03-09T13:34:54.226 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:34:54.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/5/new.json 2024-03-09T13:34:54.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9Zexl5fDWMRAAmat701broixlHMIPUuYlJA== --osd-uuid 1a7a2a5c-3d7e-4a54-91c1-c81c5be3f535 2024-03-09T13:34:54.266 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:54.265+0000 7fbe1f79e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:34:54.312 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:54.310+0000 7fbe1f79e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:34:54.312 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:54.311+0000 7fbe1f79e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:34:54.313 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:54.312+0000 7fbe1f79e640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-09T13:34:54.313 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:54.312+0000 7fbe1f79e640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-09T13:34:54.313 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:54.312+0000 7fbe1f79e640 -1 bluestore(td/osd-scrub-repair/5/block) _read_bdev_label failed to open td/osd-scrub-repair/5/block: (2) No such file or directory 2024-03-09T13:34:54.387 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:34:54.385+0000 7fbe1f79e640 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2024-03-09T13:35:00.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2024-03-09T13:35:00.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:35:00.522 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd5 key to auth repository 2024-03-09T13:35:00.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-03-09T13:35:00.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:35:00.875 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.5 2024-03-09T13:35:00.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-03-09T13:35:00.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:35:00.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:35:00.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:35:00.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:35:00.901 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:00.900+0000 7f80afe1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:35:00.903 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:00.902+0000 7f80afe1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:35:00.904 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:00.904+0000 7f80afe1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:35:01.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-03-09T13:35:01.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:35:01.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T13:35:01.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:35:01.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:35:01.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:01.156 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:35:01.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:35:01.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:01.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:01.424 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:01.423+0000 7f80afe1b640 -1 Falling back to public interface 2024-03-09T13:35:01.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:02.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:02.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:02.432 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:35:02.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:35:02.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:02.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:02.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:03.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:03.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:03.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:35:03.715 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:35:03.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:03.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:03.813 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:03.812+0000 7f80afe1b640 -1 osd.5 0 log_to_monitors true 2024-03-09T13:35:04.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:05.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:05.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:05.015 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:35:05.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:35:05.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:05.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:05.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:06.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:06.301 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:35:06.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:06.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:35:06.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:06.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:06.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:07.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:07.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:07.602 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:35:07.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:35:07.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:07.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:07.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:08.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:08.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:08.970 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:35:08.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:35:08.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:08.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:09.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:10.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:10.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:10.292 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:35:10.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:35:10.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:10.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:10.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:11.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:11.615 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:35:11.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:11.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:35:11.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:11.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:11.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:12.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:12.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:12.962 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:35:12.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:35:12.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:12.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:13.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:14.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:14.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:14.251 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:35:14.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:35:14.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:14.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:14.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:15.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:15.612 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:35:15.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:15.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:35:15.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:15.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:16.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:17.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:17.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:17.014 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:35:17.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:35:17.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:17.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:17.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:18.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:18.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:18.528 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:35:18.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:35:18.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:18.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:19.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:20.289 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:35:20.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:20.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:35:20.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:20.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:20.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:21.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:21.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:21.922 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:35:21.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:35:21.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:21.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:22.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:23.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:23.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:23.356 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:35:23.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:35:23.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:23.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:24.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:25.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:25.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:25.034 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:35:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:35:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:25.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:25.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:26.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:26.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:26.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:35:26.519 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:35:26.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:26.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:26.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:27.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:27.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:27.873 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:35:27.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:35:27.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:27.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:35:28.195 INFO:tasks.workunit.client.0.smithi162.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3263615796,v1:127.0.0.1:6843/3263615796] [v2:127.0.0.1:6844/3263615796,v1:127.0.0.1:6845/3263615796] exists,up 1a7a2a5c-3d7e-4a54-91c1-c81c5be3f535 2024-03-09T13:35:28.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:35:28.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:35:28.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:35:28.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:35:28.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/6 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:35:28.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:35:28.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:35:28.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:35:28.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:35:28.202 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:35:28.202 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:35:28.202 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:35:28.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:35:28.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:35:28.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:35:28.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:35:28.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:35:28.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:35:28.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:35:28.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:35:28.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:35:28.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:35:28.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:35:28.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:35:28.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:35:28.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/6 2024-03-09T13:35:28.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:35:28.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=caa77fb1-ef56-4e98-b6f9-8bbc9e5051fd 2024-03-09T13:35:28.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 caa77fb1-ef56-4e98-b6f9-8bbc9e5051fd' 2024-03-09T13:35:28.203 INFO:tasks.workunit.client.0.smithi162.stdout:add osd6 caa77fb1-ef56-4e98-b6f9-8bbc9e5051fd 2024-03-09T13:35:28.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:35:28.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCgZexlhab1DBAAFMknAZa8oRhxrM3LPZdiUw== 2024-03-09T13:35:28.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCgZexlhab1DBAAFMknAZa8oRhxrM3LPZdiUw=="}' 2024-03-09T13:35:28.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new caa77fb1-ef56-4e98-b6f9-8bbc9e5051fd -i td/osd-scrub-repair/6/new.json 2024-03-09T13:35:28.584 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:35:28.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/6/new.json 2024-03-09T13:35:28.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgZexlhab1DBAAFMknAZa8oRhxrM3LPZdiUw== --osd-uuid caa77fb1-ef56-4e98-b6f9-8bbc9e5051fd 2024-03-09T13:35:28.624 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:28.623+0000 7fc778c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:35:28.670 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:28.670+0000 7fc778c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:35:28.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:28.671+0000 7fc778c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:35:28.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:28.671+0000 7fc778c0d640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-09T13:35:28.672 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:28.671+0000 7fc778c0d640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-09T13:35:28.672 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:28.671+0000 7fc778c0d640 -1 bluestore(td/osd-scrub-repair/6/block) _read_bdev_label failed to open td/osd-scrub-repair/6/block: (2) No such file or directory 2024-03-09T13:35:28.743 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:28.743+0000 7fc778c0d640 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2024-03-09T13:35:35.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2024-03-09T13:35:35.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:35:35.188 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd6 key to auth repository 2024-03-09T13:35:35.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-03-09T13:35:35.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:35:35.556 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.6 2024-03-09T13:35:35.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-03-09T13:35:35.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:35:35.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:35:35.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:35:35.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:35:35.583 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:35.582+0000 7febed42a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:35:35.585 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:35.584+0000 7febed42a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:35:35.586 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:35.585+0000 7febed42a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:35:35.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-03-09T13:35:35.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:35:35.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-03-09T13:35:35.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:35:35.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:35:35.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:35.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:35:35.832 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:35:35.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:35.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:36.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:36.113 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:36.112+0000 7febed42a640 -1 Falling back to public interface 2024-03-09T13:35:37.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:37.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:37.105 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:35:37.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:35:37.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:37.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:37.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:38.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:38.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:38.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:35:38.387 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:35:38.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:38.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:38.456 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:35:38.455+0000 7febed42a640 -1 osd.6 0 log_to_monitors true 2024-03-09T13:35:38.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:39.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:39.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:39.672 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:35:39.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:35:39.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:39.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:40.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:41.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:41.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:41.064 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:35:41.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:35:41.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:41.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:41.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:42.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:42.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:42.460 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:35:42.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:35:42.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:42.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:42.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:43.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:43.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:43.745 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:35:43.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:35:43.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:43.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:44.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:45.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:45.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:45.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:35:45.084 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:35:45.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:45.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:45.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:46.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:46.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:46.616 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:35:46.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:35:46.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:46.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:46.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:47.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:47.900 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:35:47.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:47.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:35:47.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:47.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:48.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:49.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:49.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:49.322 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:35:49.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:35:49.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:49.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:49.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:50.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:50.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:50.612 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:35:50.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:35:50.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:50.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:50.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:51.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:51.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:51.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:35:51.897 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:35:51.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:51.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:52.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:53.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:53.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:53.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:35:53.637 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:35:53.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:53.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:54.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:55.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:55.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:55.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:35:55.381 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:35:55.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:55.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:56.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:57.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:57.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:57.101 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:35:57.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:35:57.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:57.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:57.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:35:58.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:35:58.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:35:58.915 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:35:58.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:35:58.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:35:58.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:35:59.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:00.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:00.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:00.732 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:36:00.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:36:00.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:00.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:36:01.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:02.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:02.469 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:36:02.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:02.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:36:02.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:02.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-03-09T13:36:02.746 INFO:tasks.workunit.client.0.smithi162.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3779623227,v1:127.0.0.1:6851/3779623227] [v2:127.0.0.1:6852/3779623227,v1:127.0.0.1:6853/3779623227] exists,up caa77fb1-ef56-4e98-b6f9-8bbc9e5051fd 2024-03-09T13:36:02.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:36:02.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:36:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:36:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:36:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2024-03-09T13:36:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:36:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:36:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-03-09T13:36:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:36:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/7 2024-03-09T13:36:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:36:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:36:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:36:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:36:02.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2024-03-09T13:36:02.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2024-03-09T13:36:02.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:36:02.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:36:02.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:36:02.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:36:02.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:36:02.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:36:02.752 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:36:02.752 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:36:02.752 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:36:02.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:36:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:36:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:36:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:36:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:36:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:36:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:36:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:36:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:36:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:36:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:36:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:36:02.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:36:02.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/7 2024-03-09T13:36:02.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:36:02.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=29501f53-305d-43ca-9e73-9462c2dd7a6f 2024-03-09T13:36:02.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 29501f53-305d-43ca-9e73-9462c2dd7a6f' 2024-03-09T13:36:02.754 INFO:tasks.workunit.client.0.smithi162.stdout:add osd7 29501f53-305d-43ca-9e73-9462c2dd7a6f 2024-03-09T13:36:02.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:36:02.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCZexlCP2zLRAA/8IXDs9J0U7l1UjHpwbmoA== 2024-03-09T13:36:02.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCZexlCP2zLRAA/8IXDs9J0U7l1UjHpwbmoA=="}' 2024-03-09T13:36:02.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 29501f53-305d-43ca-9e73-9462c2dd7a6f -i td/osd-scrub-repair/7/new.json 2024-03-09T13:36:03.097 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:36:03.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/7/new.json 2024-03-09T13:36:03.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCZexlCP2zLRAA/8IXDs9J0U7l1UjHpwbmoA== --osd-uuid 29501f53-305d-43ca-9e73-9462c2dd7a6f 2024-03-09T13:36:03.137 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:03.137+0000 7f1cb7413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:36:03.148 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:03.148+0000 7f1cb7413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:36:03.149 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:03.149+0000 7f1cb7413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:36:03.149 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:03.149+0000 7f1cb7413640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-09T13:36:03.149 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:03.149+0000 7f1cb7413640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-09T13:36:03.149 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:03.149+0000 7f1cb7413640 -1 bluestore(td/osd-scrub-repair/7/block) _read_bdev_label failed to open td/osd-scrub-repair/7/block: (2) No such file or directory 2024-03-09T13:36:03.290 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:03.290+0000 7f1cb7413640 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2024-03-09T13:36:09.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2024-03-09T13:36:09.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:36:09.208 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd7 key to auth repository 2024-03-09T13:36:09.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-03-09T13:36:09.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:36:09.573 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.7 2024-03-09T13:36:09.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-03-09T13:36:09.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:36:09.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:36:09.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:36:09.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:36:09.600 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:09.599+0000 7f11d3a12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:36:09.601 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:09.602+0000 7f11d3a12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:36:09.602 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:09.603+0000 7f11d3a12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:36:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-03-09T13:36:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:36:09.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-03-09T13:36:09.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:36:09.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:36:09.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:09.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:36:09.850 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:36:09.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:09.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:10.118 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:10.118+0000 7f11d3a12640 -1 Falling back to public interface 2024-03-09T13:36:10.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:11.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:11.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:11.125 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:36:11.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:36:11.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:11.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:11.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:12.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:12.410 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:36:12.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:12.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:36:12.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:12.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:12.451 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:12.451+0000 7f11d3a12640 -1 osd.7 0 log_to_monitors true 2024-03-09T13:36:12.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:13.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:13.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:13.701 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:36:13.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:36:13.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:13.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:13.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:14.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:14.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:14.986 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:36:14.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:36:14.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:14.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:15.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:16.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:16.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:16.408 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:36:16.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:36:16.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:16.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:16.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:17.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:17.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:17.695 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:36:17.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:36:17.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:17.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:17.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:18.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:18.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:18.980 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:36:18.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:36:18.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:18.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:19.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:20.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:20.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:20.394 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:36:20.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:36:20.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:20.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:20.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:21.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:21.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:21.681 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:36:21.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:36:21.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:21.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:21.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:22.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:22.966 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:36:22.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:22.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:36:22.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:22.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:23.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:24.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:24.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:24.389 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:36:24.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:36:24.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:24.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:24.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:25.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:25.674 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:36:25.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:25.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:36:25.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:25.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:26.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:27.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:27.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:27.297 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:36:27.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:36:27.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:27.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:27.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:28.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:28.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:28.867 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:36:28.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:36:28.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:28.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:29.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:30.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:30.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:30.346 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:36:30.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:36:30.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:30.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:30.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:31.807 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:36:31.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:36:31.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:31.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:32.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:33.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:33.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:33.726 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:36:33.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:36:33.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:33.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:34.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:35.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:35.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:35.118 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:36:35.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:36:35.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:35.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:35.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:36.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:36.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:36.406 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:36:36.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:36:36.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:36.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-03-09T13:36:36.720 INFO:tasks.workunit.client.0.smithi162.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/548999595,v1:127.0.0.1:6859/548999595] [v2:127.0.0.1:6860/548999595,v1:127.0.0.1:6861/548999595] exists,up 29501f53-305d-43ca-9e73-9462c2dd7a6f 2024-03-09T13:36:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:36:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:36:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:36:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:36:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2024-03-09T13:36:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:36:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:36:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-03-09T13:36:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:36:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/8 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:36:36.733 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:36:36.734 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:36:36.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:36:36.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:36:36.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:36:36.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:36:36.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:36:36.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:36:36.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:36:36.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:36:36.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:36:36.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:36:36.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:36:36.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:36:36.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:36:36.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/8 2024-03-09T13:36:36.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:36:36.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a79f6c3f-7eea-4800-96f5-3157db3d63dc 2024-03-09T13:36:36.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 a79f6c3f-7eea-4800-96f5-3157db3d63dc' 2024-03-09T13:36:36.735 INFO:tasks.workunit.client.0.smithi162.stdout:add osd8 a79f6c3f-7eea-4800-96f5-3157db3d63dc 2024-03-09T13:36:36.736 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:36:36.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkZexl9Bc0LBAAgE1rCDPXwd3Vj3kk7i4oZg== 2024-03-09T13:36:36.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkZexl9Bc0LBAAgE1rCDPXwd3Vj3kk7i4oZg=="}' 2024-03-09T13:36:36.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a79f6c3f-7eea-4800-96f5-3157db3d63dc -i td/osd-scrub-repair/8/new.json 2024-03-09T13:36:37.075 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:36:37.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/8/new.json 2024-03-09T13:36:37.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkZexl9Bc0LBAAgE1rCDPXwd3Vj3kk7i4oZg== --osd-uuid a79f6c3f-7eea-4800-96f5-3157db3d63dc 2024-03-09T13:36:37.115 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:37.114+0000 7f30e5a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:36:37.126 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:37.125+0000 7f30e5a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:36:37.127 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:37.126+0000 7f30e5a0f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:36:37.128 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:37.127+0000 7f30e5a0f640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-09T13:36:37.128 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:37.127+0000 7f30e5a0f640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-09T13:36:37.128 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:37.127+0000 7f30e5a0f640 -1 bluestore(td/osd-scrub-repair/8/block) _read_bdev_label failed to open td/osd-scrub-repair/8/block: (2) No such file or directory 2024-03-09T13:36:37.235 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:37.234+0000 7f30e5a0f640 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2024-03-09T13:36:43.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2024-03-09T13:36:43.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:36:43.446 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd8 key to auth repository 2024-03-09T13:36:43.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-03-09T13:36:43.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:36:43.800 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.8 2024-03-09T13:36:43.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-03-09T13:36:43.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:36:43.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:36:43.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:36:43.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:36:43.826 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:43.826+0000 7fb599011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:36:43.828 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:43.828+0000 7fb599011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:36:43.829 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:43.829+0000 7fb599011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:36:44.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-03-09T13:36:44.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:36:44.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-03-09T13:36:44.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:36:44.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:36:44.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:44.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:36:44.077 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:36:44.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:44.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:36:44.349 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:44.349+0000 7fb599011640 -1 Falling back to public interface 2024-03-09T13:36:44.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:45.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:45.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:45.352 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:36:45.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:36:45.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:45.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:36:45.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:46.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:46.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:46.639 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:36:46.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:36:46.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:46.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:36:46.690 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:36:46.690+0000 7fb599011640 -1 osd.8 0 log_to_monitors true 2024-03-09T13:36:46.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:47.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:47.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:47.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:36:47.921 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:36:47.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:47.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:36:48.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:49.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:49.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:49.209 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:36:49.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:36:49.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:49.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:36:49.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:50.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:50.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:50.524 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:36:50.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:36:50.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:50.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:36:50.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:51.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:51.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:51.810 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:36:51.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:36:51.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:51.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:36:52.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:53.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:53.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:53.156 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:36:53.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:36:53.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:53.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:36:53.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:54.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:54.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:54.508 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:36:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:36:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:54.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:36:54.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:55.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:55.800 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:36:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:36:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:55.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:36:56.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:57.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:57.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:57.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:36:57.195 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:36:57.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:57.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:36:57.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:58.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:58.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:58.485 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:36:58.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:36:58.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:58.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:36:58.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:36:59.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:36:59.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:36:59.771 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:36:59.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:36:59.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:36:59.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:37:00.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:01.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:01.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:01.428 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:37:01.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:37:01.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:01.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:37:01.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:02.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:02.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:02.887 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:37:02.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:37:02.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:02.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:37:03.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:04.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:04.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:04.531 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:37:04.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:37:04.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:04.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:37:05.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:06.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:06.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:06.230 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:37:06.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:37:06.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:06.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:37:06.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:07.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:07.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:07.848 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:37:07.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:37:07.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:07.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:37:08.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:09.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:09.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:09.332 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:37:09.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:37:09.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:09.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:37:10.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:11.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:11.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:11.095 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:37:11.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:37:11.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:11.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-03-09T13:37:11.392 INFO:tasks.workunit.client.0.smithi162.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/552760634,v1:127.0.0.1:6867/552760634] [v2:127.0.0.1:6868/552760634,v1:127.0.0.1:6869/552760634] exists,up a79f6c3f-7eea-4800-96f5-3157db3d63dc 2024-03-09T13:37:11.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:37:11.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:37:11.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:37:11.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: corrupt_and_repair_lrc: for id in $(seq 0 9) 2024-03-09T13:37:11.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2024-03-09T13:37:11.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:37:11.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:37:11.399 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:37:11.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:37:11.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:37:11.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:37:11.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:37:11.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T13:37:11.401 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:37:11.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0394dfb7-1912-4dc6-b9ea-633acdf67204 2024-03-09T13:37:11.401 INFO:tasks.workunit.client.0.smithi162.stdout:add osd9 0394dfb7-1912-4dc6-b9ea-633acdf67204 2024-03-09T13:37:11.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 0394dfb7-1912-4dc6-b9ea-633acdf67204' 2024-03-09T13:37:11.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:37:11.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHZuxlummlGBAArGK92Q74hRQP5640hGXkLw== 2024-03-09T13:37:11.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHZuxlummlGBAArGK92Q74hRQP5640hGXkLw=="}' 2024-03-09T13:37:11.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0394dfb7-1912-4dc6-b9ea-633acdf67204 -i td/osd-scrub-repair/9/new.json 2024-03-09T13:37:11.832 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:37:11.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/9/new.json 2024-03-09T13:37:11.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHZuxlummlGBAArGK92Q74hRQP5640hGXkLw== --osd-uuid 0394dfb7-1912-4dc6-b9ea-633acdf67204 2024-03-09T13:37:11.872 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:37:11.871+0000 7ff3b3138640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:37:11.933 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:37:11.932+0000 7ff3b3138640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:37:11.934 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:37:11.933+0000 7ff3b3138640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:37:11.935 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:37:11.934+0000 7ff3b3138640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-09T13:37:11.935 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:37:11.934+0000 7ff3b3138640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-09T13:37:11.935 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:37:11.934+0000 7ff3b3138640 -1 bluestore(td/osd-scrub-repair/9/block) _read_bdev_label failed to open td/osd-scrub-repair/9/block: (2) No such file or directory 2024-03-09T13:37:12.041 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:37:12.040+0000 7ff3b3138640 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2024-03-09T13:37:17.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2024-03-09T13:37:17.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:37:17.728 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd9 key to auth repository 2024-03-09T13:37:17.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-03-09T13:37:17.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:37:18.082 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.9 2024-03-09T13:37:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-03-09T13:37:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:37:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:37:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:37:18.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:37:18.108 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:37:18.107+0000 7f2d9088b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:37:18.109 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:37:18.109+0000 7f2d9088b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:37:18.110 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:37:18.110+0000 7f2d9088b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:37:18.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-03-09T13:37:18.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:37:18.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T13:37:18.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:37:18.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:37:18.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:18.358 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:37:18.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:37:18.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:18.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:18.625 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:37:18.624+0000 7f2d9088b640 -1 Falling back to public interface 2024-03-09T13:37:18.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:19.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:19.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:19.654 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:37:19.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:37:19.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:19.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:19.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:20.937 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:37:20.936+0000 7f2d9088b640 -1 osd.9 0 log_to_monitors true 2024-03-09T13:37:20.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:20.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:20.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:37:20.941 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:37:20.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:20.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:21.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:22.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:22.382 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:37:22.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:22.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:37:22.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:22.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:22.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:23.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:23.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:23.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:37:23.665 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:37:23.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:23.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:23.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:24.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:24.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:24.953 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:37:24.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:37:24.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:24.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:25.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:26.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:26.240 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:37:26.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:26.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:37:26.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:26.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:26.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:27.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:27.524 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:37:27.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:27.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:37:27.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:27.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:27.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:28.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:28.827 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:37:28.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:28.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:37:28.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:28.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:29.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:30.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:30.119 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:37:30.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:30.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:37:30.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:30.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:30.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:31.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:31.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:31.405 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:37:31.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:37:31.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:31.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:31.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:32.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:37:32.697 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:37:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:32.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:33.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:34.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:34.073 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:37:34.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:34.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:37:34.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:34.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:34.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:35.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:35.410 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:37:35.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:35.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:37:35.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:35.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:35.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:36.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:36.875 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:37:36.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:36.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:37:36.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:36.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:37.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:38.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:38.270 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:37:38.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:38.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:37:38.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:38.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:38.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:39.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:39.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:39.845 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:37:39.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:37:39.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:39.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:40.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:41.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:41.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:41.128 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:37:41.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:37:41.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:41.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:41.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:42.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:42.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:42.880 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:37:42.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:37:42.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:42.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:43.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:37:44.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:37:44.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:37:44.675 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:37:44.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:37:44.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:37:44.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:37:44.964 INFO:tasks.workunit.client.0.smithi162.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2035638811,v1:127.0.0.1:6875/2035638811] [v2:127.0.0.1:6876/2035638811,v1:127.0.0.1:6877/2035638811] exists,up 0394dfb7-1912-4dc6-b9ea-633acdf67204 2024-03-09T13:37:44.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:37:44.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:37:44.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:37:44.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:834: corrupt_and_repair_lrc: create_rbd_pool 2024-03-09T13:37:44.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T13:37:45.274 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T13:37:45.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T13:37:45.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T13:37:45.809 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T13:37:45.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:37:46.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T13:37:50.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: corrupt_and_repair_lrc: wait_for_clean 2024-03-09T13:37:50.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:37:50.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:37:50.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:37:50.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:37:50.246 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:37:50.246 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:37:50.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:37:50.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:37:50.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:37:50.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:37:50.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:37:50.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:37:50.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:37:50.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:37:50.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:37:50.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:37:50.661 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:37:50.661 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:37:50.661 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:37:50.661 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-03-09T13:37:50.662 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:37:50.662 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:37:50.662 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-03-09T13:37:50.662 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-03-09T13:37:50.662 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-03-09T13:37:50.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:37:50.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:37:50.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:37:50.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836549 2024-03-09T13:37:50.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836549 2024-03-09T13:37:50.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549' 2024-03-09T13:37:50.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:37:50.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:37:50.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673022 2024-03-09T13:37:50.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673022 2024-03-09T13:37:50.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673022' 2024-03-09T13:37:50.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:37:50.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:37:51.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509491 2024-03-09T13:37:51.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509491 2024-03-09T13:37:51.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673022 2-64424509491' 2024-03-09T13:37:51.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:37:51.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:37:51.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345965 2024-03-09T13:37:51.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345965 2024-03-09T13:37:51.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673022 2-64424509491 3-85899345965' 2024-03-09T13:37:51.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:37:51.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:37:51.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182445 2024-03-09T13:37:51.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182445 2024-03-09T13:37:51.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673022 2-64424509491 3-85899345965 4-107374182445' 2024-03-09T13:37:51.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:37:51.304 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:37:51.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018914 2024-03-09T13:37:51.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018914 2024-03-09T13:37:51.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673022 2-64424509491 3-85899345965 4-107374182445 5-128849018914' 2024-03-09T13:37:51.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:37:51.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:37:51.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855390 2024-03-09T13:37:51.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855390 2024-03-09T13:37:51.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673022 2-64424509491 3-85899345965 4-107374182445 5-128849018914 6-150323855390' 2024-03-09T13:37:51.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:37:51.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:37:51.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691862 2024-03-09T13:37:51.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691862 2024-03-09T13:37:51.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673022 2-64424509491 3-85899345965 4-107374182445 5-128849018914 6-150323855390 7-171798691862' 2024-03-09T13:37:51.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:37:51.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:37:51.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528339 2024-03-09T13:37:51.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528339 2024-03-09T13:37:51.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673022 2-64424509491 3-85899345965 4-107374182445 5-128849018914 6-150323855390 7-171798691862 8-193273528339' 2024-03-09T13:37:51.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:37:51.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:37:51.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364810 2024-03-09T13:37:51.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364810 2024-03-09T13:37:51.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836549 1-42949673022 2-64424509491 3-85899345965 4-107374182445 5-128849018914 6-150323855390 7-171798691862 8-193273528339 9-214748364810' 2024-03-09T13:37:51.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:37:51.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836549 2024-03-09T13:37:51.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:37:51.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:37:51.827 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836549 2024-03-09T13:37:51.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:37:51.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836549 2024-03-09T13:37:51.829 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836549 2024-03-09T13:37:51.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836549' 2024-03-09T13:37:51.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:37:52.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836549 -lt 21474836549 2024-03-09T13:37:52.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:37:52.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673022 2024-03-09T13:37:52.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:37:52.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:37:52.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673022 2024-03-09T13:37:52.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:37:52.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673022 2024-03-09T13:37:52.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673022' 2024-03-09T13:37:52.106 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673022 2024-03-09T13:37:52.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:37:52.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673022 -lt 42949673022 2024-03-09T13:37:52.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:37:52.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509491 2024-03-09T13:37:52.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:37:52.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:37:52.383 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509491 2024-03-09T13:37:52.383 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:37:52.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509491 2024-03-09T13:37:52.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509491' 2024-03-09T13:37:52.384 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509491 2024-03-09T13:37:52.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:37:52.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509492 -lt 64424509491 2024-03-09T13:37:52.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:37:52.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345965 2024-03-09T13:37:52.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:37:52.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:37:52.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345965 2024-03-09T13:37:52.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:37:52.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345965 2024-03-09T13:37:52.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345965' 2024-03-09T13:37:52.663 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345965 2024-03-09T13:37:52.663 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:37:52.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345965 -lt 85899345965 2024-03-09T13:37:52.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:37:52.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182445 2024-03-09T13:37:52.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:37:52.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:37:52.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182445 2024-03-09T13:37:52.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:37:52.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182445 2024-03-09T13:37:52.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182445' 2024-03-09T13:37:52.943 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182445 2024-03-09T13:37:52.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:37:53.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182444 -lt 107374182445 2024-03-09T13:37:53.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:37:54.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:37:54.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:37:54.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182445 -lt 107374182445 2024-03-09T13:37:54.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:37:54.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018914 2024-03-09T13:37:54.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:37:54.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:37:54.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018914 2024-03-09T13:37:54.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:37:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018914 2024-03-09T13:37:54.504 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 128849018914 2024-03-09T13:37:54.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018914' 2024-03-09T13:37:54.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:37:54.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018914 -lt 128849018914 2024-03-09T13:37:54.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:37:54.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855390 2024-03-09T13:37:54.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:37:54.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:37:54.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855390 2024-03-09T13:37:54.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:37:54.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855390 2024-03-09T13:37:54.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855390' 2024-03-09T13:37:54.784 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 150323855390 2024-03-09T13:37:54.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:37:55.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855391 -lt 150323855390 2024-03-09T13:37:55.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:37:55.060 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691862 2024-03-09T13:37:55.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:37:55.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:37:55.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691862 2024-03-09T13:37:55.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:37:55.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691862 2024-03-09T13:37:55.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691862' 2024-03-09T13:37:55.063 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691862 2024-03-09T13:37:55.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:37:55.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691862 -lt 171798691862 2024-03-09T13:37:55.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:37:55.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528339 2024-03-09T13:37:55.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:37:55.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:37:55.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528339 2024-03-09T13:37:55.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:37:55.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528339 2024-03-09T13:37:55.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528339' 2024-03-09T13:37:55.380 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528339 2024-03-09T13:37:55.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:37:55.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528339 -lt 193273528339 2024-03-09T13:37:55.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:37:55.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364810 2024-03-09T13:37:55.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:37:55.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:37:55.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364810 2024-03-09T13:37:55.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:37:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364810 2024-03-09T13:37:55.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364810' 2024-03-09T13:37:55.696 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 214748364810 2024-03-09T13:37:55.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:37:56.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364811 -lt 214748364810 2024-03-09T13:37:56.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:37:56.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:37:56.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:37:56.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T13:37:56.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:37:56.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:37:56.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:37:56.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:37:56.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:37:56.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:37:56.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:37:56.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:37:56.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:37:56.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:37:56.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:37:57.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T13:37:57.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:37:57.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:37:57.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2024-03-09T13:37:57.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T13:37:57.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T13:37:57.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-09T13:37:57.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T13:37:57.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2024-03-09T13:37:57.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T13:37:57.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T13:37:58.138 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' created 2024-03-09T13:37:58.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:37:59.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-09T13:37:59.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-09T13:37:59.546 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 2 allow_ec_overwrites to true 2024-03-09T13:37:59.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T13:37:59.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:37:59.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:37:59.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:37:59.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:37:59.559 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:37:59.559 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:37:59.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:37:59.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:37:59.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:37:59.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:37:59.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:37:59.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:37:59.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:37:59.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:37:59.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:37:59.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:37:59.954 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:37:59.954 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:37:59.954 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:37:59.954 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-03-09T13:37:59.954 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:37:59.954 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:37:59.954 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-03-09T13:37:59.955 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-03-09T13:37:59.955 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-03-09T13:37:59.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:37:59.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:37:59.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:38:00.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836552 2024-03-09T13:38:00.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836552 2024-03-09T13:38:00.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552' 2024-03-09T13:38:00.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:00.151 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:38:00.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673025 2024-03-09T13:38:00.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673025 2024-03-09T13:38:00.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-42949673025' 2024-03-09T13:38:00.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:00.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:38:00.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509494 2024-03-09T13:38:00.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509494 2024-03-09T13:38:00.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-42949673025 2-64424509494' 2024-03-09T13:38:00.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:00.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:38:00.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345968 2024-03-09T13:38:00.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345968 2024-03-09T13:38:00.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-42949673025 2-64424509494 3-85899345968' 2024-03-09T13:38:00.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:00.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:38:00.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182448 2024-03-09T13:38:00.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182448 2024-03-09T13:38:00.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-42949673025 2-64424509494 3-85899345968 4-107374182448' 2024-03-09T13:38:00.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:00.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:38:00.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018917 2024-03-09T13:38:00.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018917 2024-03-09T13:38:00.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-42949673025 2-64424509494 3-85899345968 4-107374182448 5-128849018917' 2024-03-09T13:38:00.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:00.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:38:00.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855393 2024-03-09T13:38:00.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855393 2024-03-09T13:38:00.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-42949673025 2-64424509494 3-85899345968 4-107374182448 5-128849018917 6-150323855393' 2024-03-09T13:38:00.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:00.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:38:01.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691865 2024-03-09T13:38:01.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691865 2024-03-09T13:38:01.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-42949673025 2-64424509494 3-85899345968 4-107374182448 5-128849018917 6-150323855393 7-171798691865' 2024-03-09T13:38:01.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:01.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:38:01.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528341 2024-03-09T13:38:01.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528341 2024-03-09T13:38:01.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-42949673025 2-64424509494 3-85899345968 4-107374182448 5-128849018917 6-150323855393 7-171798691865 8-193273528341' 2024-03-09T13:38:01.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:01.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:38:01.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364813 2024-03-09T13:38:01.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364813 2024-03-09T13:38:01.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836552 1-42949673025 2-64424509494 3-85899345968 4-107374182448 5-128849018917 6-150323855393 7-171798691865 8-193273528341 9-214748364813' 2024-03-09T13:38:01.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:01.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836552 2024-03-09T13:38:01.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:01.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:38:01.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836552 2024-03-09T13:38:01.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:01.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836552 2024-03-09T13:38:01.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836552' 2024-03-09T13:38:01.304 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836552 2024-03-09T13:38:01.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:38:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836552 -lt 21474836552 2024-03-09T13:38:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:01.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673025 2024-03-09T13:38:01.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:01.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:38:01.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673025 2024-03-09T13:38:01.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:01.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673025 2024-03-09T13:38:01.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673025' 2024-03-09T13:38:01.608 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949673025 2024-03-09T13:38:01.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:38:01.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673025 -lt 42949673025 2024-03-09T13:38:01.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:01.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509494 2024-03-09T13:38:01.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:01.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:38:01.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509494 2024-03-09T13:38:01.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:01.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509494 2024-03-09T13:38:01.885 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509494 2024-03-09T13:38:01.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509494' 2024-03-09T13:38:01.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:38:02.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509495 -lt 64424509494 2024-03-09T13:38:02.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:02.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345968 2024-03-09T13:38:02.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:02.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:38:02.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345968 2024-03-09T13:38:02.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:02.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345968 2024-03-09T13:38:02.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345968' 2024-03-09T13:38:02.162 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.3 seq 85899345968 2024-03-09T13:38:02.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:38:02.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345968 -lt 85899345968 2024-03-09T13:38:02.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:02.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182448 2024-03-09T13:38:02.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:02.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:38:02.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182448 2024-03-09T13:38:02.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:02.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182448 2024-03-09T13:38:02.438 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.4 seq 107374182448 2024-03-09T13:38:02.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182448' 2024-03-09T13:38:02.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:38:02.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182448 -lt 107374182448 2024-03-09T13:38:02.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:02.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018917 2024-03-09T13:38:02.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:02.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:38:02.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018917 2024-03-09T13:38:02.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:02.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018917 2024-03-09T13:38:02.714 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.5 seq 128849018917 2024-03-09T13:38:02.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018917' 2024-03-09T13:38:02.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:38:02.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018917 -lt 128849018917 2024-03-09T13:38:02.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:02.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855393 2024-03-09T13:38:02.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:02.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:38:02.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855393 2024-03-09T13:38:02.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:02.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855393 2024-03-09T13:38:02.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855393' 2024-03-09T13:38:02.994 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.6 seq 150323855393 2024-03-09T13:38:02.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:38:03.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855393 -lt 150323855393 2024-03-09T13:38:03.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:03.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691865 2024-03-09T13:38:03.360 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:03.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:38:03.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691865 2024-03-09T13:38:03.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:03.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691865 2024-03-09T13:38:03.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691865' 2024-03-09T13:38:03.363 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.7 seq 171798691865 2024-03-09T13:38:03.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:38:03.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691865 -lt 171798691865 2024-03-09T13:38:03.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:03.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528341 2024-03-09T13:38:03.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:03.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:38:03.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528341 2024-03-09T13:38:03.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:03.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528341 2024-03-09T13:38:03.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528341' 2024-03-09T13:38:03.641 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.8 seq 193273528341 2024-03-09T13:38:03.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:38:03.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528342 -lt 193273528341 2024-03-09T13:38:03.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:03.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364813 2024-03-09T13:38:03.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:03.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:38:03.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364813 2024-03-09T13:38:03.920 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:03.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364813 2024-03-09T13:38:03.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364813' 2024-03-09T13:38:03.922 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.9 seq 214748364813 2024-03-09T13:38:03.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:38:04.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364813 -lt 214748364813 2024-03-09T13:38:04.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:38:04.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:38:04.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:38:04.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:38:04.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:38:04.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:38:04.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:38:04.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:38:04.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:38:04.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:38:04.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:38:04.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:38:04.811 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:38:04.811 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:38:04.811 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:38:05.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:38:05.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:38:05.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:38:05.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T13:38:05.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:838: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2024-03-09T13:38:05.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T13:38:05.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:249: corrupt_and_repair_erasure_coded: local poolname=ecpool 2024-03-09T13:38:05.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:251: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2024-03-09T13:38:05.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:38:05.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:38:05.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-09T13:38:05.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:38:05.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:38:05.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:38:05.630 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:38:05.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:38:06.150 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:38:06.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:38:06.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:38:06.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T13:38:06.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2024-03-09T13:38:06.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=ecpool 2024-03-09T13:38:06.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:38:06.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:38:06.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:38:06.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_erasure_coded: local primary=3 2024-03-09T13:38:06.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2024-03-09T13:38:06.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T13:38:06.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-03-09T13:38:06.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: sed -e s/3// 2024-03-09T13:38:06.963 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:38:06.964 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T13:38:07.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-03-09T13:38:07.240 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:38:07.240 INFO:tasks.workunit.client.0.smithi162.stderr:9 2024-03-09T13:38:07.240 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:38:07.240 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:38:07.240 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:38:07.240 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:38:07.240 INFO:tasks.workunit.client.0.smithi162.stderr:7' 2024-03-09T13:38:07.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 6 2 1 7 2024-03-09T13:38:07.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2024-03-09T13:38:07.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_erasure_coded: local -a osds 2024-03-09T13:38:07.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_erasure_coded: local not_primary_first=5 2024-03-09T13:38:07.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_erasure_coded: local not_primary_second=9 2024-03-09T13:38:07.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:259: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2024-03-09T13:38:07.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=3 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:38:07.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:38:07.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:38:07.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:38:07.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:38:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:38:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:38:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:38:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:38:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:38:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:38:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:38:07.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T13:38:09.384 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T13:38:09.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:38:09.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:38:09.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:38:09.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:38:09.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:38:09.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:38:09.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:38:09.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:38:09.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:38:09.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:38:09.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:38:09.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:38:09.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:38:09.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:38:09.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:38:09.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:38:09.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:38:09.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:38:09.903 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:38:09.903 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:38:09.903 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:38:09.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:38:09.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:38:09.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:38:09.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:38:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:38:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:38:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:38:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:38:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:38:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:38:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:38:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:38:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:38:09.904 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.3 2024-03-09T13:38:09.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:38:09.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:38:09.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:38:09.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:38:09.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:38:09.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:38:09.931 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:09.930+0000 7f5a8b264640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:38:09.933 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:09.932+0000 7f5a8b264640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:38:09.935 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:09.934+0000 7f5a8b264640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:38:10.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:38:10.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:38:10.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:38:10.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:38:10.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:38:10.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:10.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:38:10.190 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:38:10.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:10.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:38:10.449 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:10.448+0000 7f5a8b264640 -1 Falling back to public interface 2024-03-09T13:38:10.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:38:11.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:38:11.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:11.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:38:11.467 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:38:11.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:11.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:38:11.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:38:12.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:38:12.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:12.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:38:12.749 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:38:12.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:12.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:38:12.787 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:12.786+0000 7f5a8b264640 -1 osd.3 69 log_to_monitors true 2024-03-09T13:38:13.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:38:14.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:38:14.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:14.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:38:14.032 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:38:14.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:14.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:38:14.443 INFO:tasks.workunit.client.0.smithi162.stderr:osd.3 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [20,69) [v2:127.0.0.1:6826/1860235877,v1:127.0.0.1:6827/1860235877] [v2:127.0.0.1:6828/1860235877,v1:127.0.0.1:6829/1860235877] exists,up 2095e0ae-504c-4658-8991-61158f227389 2024-03-09T13:38:14.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:38:14.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:38:14.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:38:14.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:38:14.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:38:14.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:38:14.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:38:14.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:38:14.444 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:38:14.444 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:38:14.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:38:14.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:38:14.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:38:14.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:38:14.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:38:14.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:38:14.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:38:14.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:38:14.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:38:14.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:38:14.831 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:38:14.831 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:38:14.831 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:38:14.831 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-03-09T13:38:14.831 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:38:14.831 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:38:14.831 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-03-09T13:38:14.832 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-03-09T13:38:14.832 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-03-09T13:38:14.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:38:14.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:14.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:38:14.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836556 2024-03-09T13:38:14.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836556 2024-03-09T13:38:14.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556' 2024-03-09T13:38:14.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:14.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:38:15.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673029 2024-03-09T13:38:15.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673029 2024-03-09T13:38:15.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029' 2024-03-09T13:38:15.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:15.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:38:15.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509498 2024-03-09T13:38:15.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509498 2024-03-09T13:38:15.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498' 2024-03-09T13:38:15.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:15.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:38:15.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-03-09T13:38:15.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-03-09T13:38:15.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-313532612610' 2024-03-09T13:38:15.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:15.314 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:38:15.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182452 2024-03-09T13:38:15.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182452 2024-03-09T13:38:15.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-313532612610 4-107374182452' 2024-03-09T13:38:15.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:15.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:38:15.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018921 2024-03-09T13:38:15.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018921 2024-03-09T13:38:15.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-313532612610 4-107374182452 5-128849018921' 2024-03-09T13:38:15.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:15.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:38:15.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855397 2024-03-09T13:38:15.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855397 2024-03-09T13:38:15.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-313532612610 4-107374182452 5-128849018921 6-150323855397' 2024-03-09T13:38:15.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:15.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:38:15.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691868 2024-03-09T13:38:15.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691868 2024-03-09T13:38:15.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-313532612610 4-107374182452 5-128849018921 6-150323855397 7-171798691868' 2024-03-09T13:38:15.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:15.744 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:38:15.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528345 2024-03-09T13:38:15.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528345 2024-03-09T13:38:15.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-313532612610 4-107374182452 5-128849018921 6-150323855397 7-171798691868 8-193273528345' 2024-03-09T13:38:15.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:15.850 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:38:15.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364817 2024-03-09T13:38:15.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364817 2024-03-09T13:38:15.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836556 1-42949673029 2-64424509498 3-313532612610 4-107374182452 5-128849018921 6-150323855397 7-171798691868 8-193273528345 9-214748364817' 2024-03-09T13:38:15.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:15.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836556 2024-03-09T13:38:15.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:15.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:38:15.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836556 2024-03-09T13:38:15.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:15.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836556 2024-03-09T13:38:15.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836556' 2024-03-09T13:38:15.957 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836556 2024-03-09T13:38:15.957 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:38:16.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836556 -lt 21474836556 2024-03-09T13:38:16.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:16.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673029 2024-03-09T13:38:16.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:16.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:38:16.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673029 2024-03-09T13:38:16.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:16.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673029 2024-03-09T13:38:16.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673029' 2024-03-09T13:38:16.232 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673029 2024-03-09T13:38:16.233 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:38:16.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673029 -lt 42949673029 2024-03-09T13:38:16.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:16.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509498 2024-03-09T13:38:16.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:16.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:38:16.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509498 2024-03-09T13:38:16.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:16.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509498 2024-03-09T13:38:16.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2024-03-09T13:38:16.510 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509498 2024-03-09T13:38:16.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:38:16.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509498 -lt 64424509498 2024-03-09T13:38:16.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:16.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612610 2024-03-09T13:38:16.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:16.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:38:16.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612610 2024-03-09T13:38:16.788 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:16.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-03-09T13:38:16.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612610' 2024-03-09T13:38:16.789 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 313532612610 2024-03-09T13:38:16.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:38:17.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 313532612610 2024-03-09T13:38:17.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:38:18.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:38:18.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:38:18.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612610 2024-03-09T13:38:18.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:18.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182452 2024-03-09T13:38:18.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:18.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:38:18.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182452 2024-03-09T13:38:18.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:18.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182452 2024-03-09T13:38:18.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182452' 2024-03-09T13:38:18.351 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182452 2024-03-09T13:38:18.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:38:18.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182452 -lt 107374182452 2024-03-09T13:38:18.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:18.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018921 2024-03-09T13:38:18.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:18.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:38:18.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018921 2024-03-09T13:38:18.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018921 2024-03-09T13:38:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018921' 2024-03-09T13:38:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 128849018921 2024-03-09T13:38:18.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:38:18.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018921 -lt 128849018921 2024-03-09T13:38:18.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:18.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855397 2024-03-09T13:38:18.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:18.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:38:18.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855397 2024-03-09T13:38:18.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:18.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855397 2024-03-09T13:38:18.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855397' 2024-03-09T13:38:18.910 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 150323855397 2024-03-09T13:38:18.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:38:19.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855398 -lt 150323855397 2024-03-09T13:38:19.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:19.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691868 2024-03-09T13:38:19.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:19.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:38:19.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691868 2024-03-09T13:38:19.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:19.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691868 2024-03-09T13:38:19.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2024-03-09T13:38:19.186 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691868 2024-03-09T13:38:19.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:38:19.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691869 -lt 171798691868 2024-03-09T13:38:19.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:19.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528345 2024-03-09T13:38:19.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:19.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:38:19.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528345 2024-03-09T13:38:19.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:19.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528345 2024-03-09T13:38:19.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-03-09T13:38:19.467 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528345 2024-03-09T13:38:19.468 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:38:19.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528346 -lt 193273528345 2024-03-09T13:38:19.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:19.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364817 2024-03-09T13:38:19.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:19.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:38:19.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364817 2024-03-09T13:38:19.763 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:19.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364817 2024-03-09T13:38:19.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364817' 2024-03-09T13:38:19.765 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 214748364817 2024-03-09T13:38:19.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:38:20.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364817 -lt 214748364817 2024-03-09T13:38:20.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:38:20.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:38:20.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:38:20.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:38:20.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:38:20.515 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:38:20.515 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:38:20.515 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:38:20.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:38:20.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:38:20.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:38:20.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:38:20.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:38:20.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:38:20.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:38:21.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:38:21.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:38:21.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:38:21.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T13:38:21.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:38:21.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:38:21.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:38:21.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:38:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T13:38:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T13:38:21.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:38:21.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:38:21.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:38:21.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:38:21.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:38:21.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:38:21.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:37:58.004094+0000 2024-03-09T13:38:21.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:38:21.942 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:38:21.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:37:58.004094+0000 2024-03-09T13:38:21.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:38:21.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:37:58.004094+0000 2024-03-09T13:38:21.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:38:21.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:38:21.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:38:21.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:38:21.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:38:21.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:38:21.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:38:21.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:38:22.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:37:58.004094+0000 '>' 2024-03-09T13:37:58.004094+0000 2024-03-09T13:38:22.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:38:23.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:38:23.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:38:23.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:38:23.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:38:23.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:38:23.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:38:23.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:38:23.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:38:22.877790+0000 '>' 2024-03-09T13:37:58.004094+0000 2024-03-09T13:38:23.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:38:23.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:38:23.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:38:23.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:38:23.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T13:38:23.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:38:23.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:38:23.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:38:23.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:38:23.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T13:38:23.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:38:23.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T13:38:23.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:38:23.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:38:23.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:38:23.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:38:23.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:38:23.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:38:23.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:38:23.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:38:23.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:38:23.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T13:38:23.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:38:23.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:38:23.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-09T13:38:25.928 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:38:25.929 INFO:tasks.workunit.client.0.smithi162.stdout:hinfo_key 2024-03-09T13:38:25.929 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:38:26.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T13:38:26.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:38:26.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:38:26.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T13:38:26.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:38:26.198 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T13:38:26.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T13:38:26.200 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.9 2024-03-09T13:38:26.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:38:26.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T13:38:26.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T13:38:26.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:38:26.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:38:26.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:38:26.227 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:26.227+0000 7fbd33410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:38:26.230 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:26.230+0000 7fbd33410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:38:26.231 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:26.231+0000 7fbd33410640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:38:26.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T13:38:26.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:38:26.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T13:38:26.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:38:26.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:38:26.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:26.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:38:26.483 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:38:26.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:26.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:38:26.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:38:26.759 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:26.760+0000 7fbd33410640 -1 Falling back to public interface 2024-03-09T13:38:27.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:38:27.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:27.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:38:27.760 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:38:27.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:27.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:38:28.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:38:29.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:38:29.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:29.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:38:29.045 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:38:29.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:29.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:38:29.075 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:29.075+0000 7fbd33410640 -1 osd.9 74 log_to_monitors true 2024-03-09T13:38:29.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:38:30.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:38:30.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:30.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:38:30.424 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:38:30.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:30.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:38:31.176 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 up in weight 1 up_from 78 up_thru 0 down_at 75 last_clean_interval [50,74) [v2:127.0.0.1:6874/2778093890,v1:127.0.0.1:6875/2778093890] [v2:127.0.0.1:6876/2778093890,v1:127.0.0.1:6877/2778093890] exists,up 0394dfb7-1912-4dc6-b9ea-633acdf67204 2024-03-09T13:38:31.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:38:31.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:38:31.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:38:31.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:38:31.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:38:31.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:38:31.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:38:31.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:38:31.177 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:38:31.177 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:38:31.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:38:31.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:38:31.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:38:31.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:38:31.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:38:31.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:38:31.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:38:31.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:38:31.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:38:31.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-03-09T13:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-03-09T13:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-03-09T13:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-03-09T13:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:31.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:38:31.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836560 2024-03-09T13:38:31.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836560 2024-03-09T13:38:31.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560' 2024-03-09T13:38:31.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:31.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:38:32.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673033 2024-03-09T13:38:32.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673033 2024-03-09T13:38:32.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033' 2024-03-09T13:38:32.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:32.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:38:32.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509503 2024-03-09T13:38:32.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509503 2024-03-09T13:38:32.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509503' 2024-03-09T13:38:32.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:32.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:38:32.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612614 2024-03-09T13:38:32.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612614 2024-03-09T13:38:32.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509503 3-313532612614' 2024-03-09T13:38:32.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:32.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:38:32.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182456 2024-03-09T13:38:32.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182456 2024-03-09T13:38:32.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509503 3-313532612614 4-107374182456' 2024-03-09T13:38:32.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:32.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:38:32.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018925 2024-03-09T13:38:32.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018925 2024-03-09T13:38:32.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509503 3-313532612614 4-107374182456 5-128849018925' 2024-03-09T13:38:32.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:32.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:38:32.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855402 2024-03-09T13:38:32.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855402 2024-03-09T13:38:32.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509503 3-313532612614 4-107374182456 5-128849018925 6-150323855402' 2024-03-09T13:38:32.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:32.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:38:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691873 2024-03-09T13:38:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691873 2024-03-09T13:38:32.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509503 3-313532612614 4-107374182456 5-128849018925 6-150323855402 7-171798691873' 2024-03-09T13:38:32.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:32.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:38:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528350 2024-03-09T13:38:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528350 2024-03-09T13:38:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509503 3-313532612614 4-107374182456 5-128849018925 6-150323855402 7-171798691873 8-193273528350' 2024-03-09T13:38:32.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:32.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:38:32.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449090 2024-03-09T13:38:32.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449090 2024-03-09T13:38:32.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836560 1-42949673033 2-64424509503 3-313532612614 4-107374182456 5-128849018925 6-150323855402 7-171798691873 8-193273528350 9-335007449090' 2024-03-09T13:38:32.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:32.849 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836560 2024-03-09T13:38:32.849 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:32.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:38:32.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836560 2024-03-09T13:38:32.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:32.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836560 2024-03-09T13:38:32.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836560' 2024-03-09T13:38:32.852 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836560 2024-03-09T13:38:32.852 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:38:33.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836559 -lt 21474836560 2024-03-09T13:38:33.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:38:34.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:38:34.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:38:34.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836560 -lt 21474836560 2024-03-09T13:38:34.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:34.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673033 2024-03-09T13:38:34.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:34.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:38:34.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673033 2024-03-09T13:38:34.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:34.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673033 2024-03-09T13:38:34.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673033' 2024-03-09T13:38:34.414 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673033 2024-03-09T13:38:34.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:38:34.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673033 -lt 42949673033 2024-03-09T13:38:34.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:34.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509503 2024-03-09T13:38:34.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:34.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:38:34.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509503 2024-03-09T13:38:34.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:34.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509503 2024-03-09T13:38:34.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509503' 2024-03-09T13:38:34.692 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509503 2024-03-09T13:38:34.693 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:38:34.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509503 -lt 64424509503 2024-03-09T13:38:34.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:34.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612614 2024-03-09T13:38:34.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:34.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:38:34.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612614 2024-03-09T13:38:34.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:34.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612614 2024-03-09T13:38:34.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612614' 2024-03-09T13:38:34.970 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 313532612614 2024-03-09T13:38:34.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:38:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612615 -lt 313532612614 2024-03-09T13:38:35.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:35.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182456 2024-03-09T13:38:35.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:35.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:38:35.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182456 2024-03-09T13:38:35.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:35.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182456 2024-03-09T13:38:35.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182456' 2024-03-09T13:38:35.248 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182456 2024-03-09T13:38:35.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:38:35.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182457 -lt 107374182456 2024-03-09T13:38:35.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:35.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018925 2024-03-09T13:38:35.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:35.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:38:35.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018925 2024-03-09T13:38:35.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:35.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018925 2024-03-09T13:38:35.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018925' 2024-03-09T13:38:35.534 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 128849018925 2024-03-09T13:38:35.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:38:35.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018926 -lt 128849018925 2024-03-09T13:38:35.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:35.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855402 2024-03-09T13:38:35.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:35.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:38:35.811 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855402 2024-03-09T13:38:35.811 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:35.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855402 2024-03-09T13:38:35.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855402' 2024-03-09T13:38:35.812 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 150323855402 2024-03-09T13:38:35.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:38:36.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855402 -lt 150323855402 2024-03-09T13:38:36.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:36.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691873 2024-03-09T13:38:36.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:36.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:38:36.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691873 2024-03-09T13:38:36.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:36.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691873 2024-03-09T13:38:36.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2024-03-09T13:38:36.134 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691873 2024-03-09T13:38:36.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:38:36.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691873 -lt 171798691873 2024-03-09T13:38:36.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:36.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528350 2024-03-09T13:38:36.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:36.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:38:36.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528350 2024-03-09T13:38:36.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:36.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528350 2024-03-09T13:38:36.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528350' 2024-03-09T13:38:36.417 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528350 2024-03-09T13:38:36.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:38:36.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528350 -lt 193273528350 2024-03-09T13:38:36.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:36.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-335007449090 2024-03-09T13:38:36.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:36.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:38:36.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-335007449090 2024-03-09T13:38:36.692 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:36.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449090 2024-03-09T13:38:36.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 335007449090' 2024-03-09T13:38:36.693 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 335007449090 2024-03-09T13:38:36.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:38:36.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449091 -lt 335007449090 2024-03-09T13:38:36.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:38:36.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:38:36.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:38:37.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:38:37.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:38:37.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:38:37.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:38:37.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:38:37.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:38:37.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:38:37.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:38:37.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:38:37.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:38:37.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:38:37.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:38:37.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:38:37.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:38:37.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:38:37.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:38:37.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:38:37.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:261: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2024-03-09T13:38:37.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T13:38:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=ecpool 2024-03-09T13:38:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=5 2024-03-09T13:38:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:38:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:38:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:38:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T13:38:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:38:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:38:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:38:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:38:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T13:38:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:38:37.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T13:38:37.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:38:37.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:38:37.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:38:37.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:38:37.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:38:38.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:38:38.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:38:38.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:38:38.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:38:38.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T13:38:38.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:38:38.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:38:38.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-09T13:38:40.233 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:38:40.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T13:38:40.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:38:40.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:38:40.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:38:40.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:38:40.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T13:38:40.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T13:38:40.755 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.5 2024-03-09T13:38:40.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:38:40.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T13:38:40.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T13:38:40.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:38:40.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:38:40.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:38:40.781 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:40.780+0000 7f80c18ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:38:40.782 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:40.782+0000 7f80c18ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:38:40.783 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:40.783+0000 7f80c18ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:38:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T13:38:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:38:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T13:38:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:38:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:38:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:38:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:38:41.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:41.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:38:41.305 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:41.305+0000 7f80c18ce640 -1 Falling back to public interface 2024-03-09T13:38:41.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:38:42.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:38:42.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:42.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:38:42.320 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:38:42.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:42.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:38:42.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:38:43.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:38:43.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:43.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:38:43.606 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:38:43.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:43.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:38:43.812 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:43.811+0000 7f80c18ce640 -1 osd.5 80 log_to_monitors true 2024-03-09T13:38:43.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:38:44.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:38:44.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:44.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:38:44.918 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:38:44.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:44.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:38:45.260 INFO:tasks.workunit.client.0.smithi162.stderr:osd.5 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [30,80) [v2:127.0.0.1:6842/171564718,v1:127.0.0.1:6843/171564718] [v2:127.0.0.1:6844/171564718,v1:127.0.0.1:6845/171564718] exists,up 1a7a2a5c-3d7e-4a54-91c1-c81c5be3f535 2024-03-09T13:38:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:38:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:38:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:38:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:38:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:38:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:38:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:38:45.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:38:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:38:45.261 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:38:45.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:38:45.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:38:45.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:38:45.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:38:45.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:38:45.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:38:45.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:38:45.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:38:45.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:38:45.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:38:45.823 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:38:45.823 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:38:45.823 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:38:45.823 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-03-09T13:38:45.823 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:38:45.823 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:38:45.823 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-03-09T13:38:45.823 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-03-09T13:38:45.823 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-03-09T13:38:45.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:38:45.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:45.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:38:45.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836564 2024-03-09T13:38:45.938 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836564 2024-03-09T13:38:45.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564' 2024-03-09T13:38:45.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:45.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:38:46.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673037 2024-03-09T13:38:46.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673037 2024-03-09T13:38:46.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037' 2024-03-09T13:38:46.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:46.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:38:46.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509507 2024-03-09T13:38:46.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509507 2024-03-09T13:38:46.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509507' 2024-03-09T13:38:46.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:46.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:38:46.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612618 2024-03-09T13:38:46.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612618 2024-03-09T13:38:46.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509507 3-313532612618' 2024-03-09T13:38:46.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:46.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:38:46.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182460 2024-03-09T13:38:46.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182460 2024-03-09T13:38:46.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509507 3-313532612618 4-107374182460' 2024-03-09T13:38:46.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:46.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:38:46.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252866 2024-03-09T13:38:46.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252866 2024-03-09T13:38:46.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509507 3-313532612618 4-107374182460 5-360777252866' 2024-03-09T13:38:46.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:46.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:38:46.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855405 2024-03-09T13:38:46.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855405 2024-03-09T13:38:46.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509507 3-313532612618 4-107374182460 5-360777252866 6-150323855405' 2024-03-09T13:38:46.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:46.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:38:46.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691877 2024-03-09T13:38:46.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691877 2024-03-09T13:38:46.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509507 3-313532612618 4-107374182460 5-360777252866 6-150323855405 7-171798691877' 2024-03-09T13:38:46.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:46.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:38:46.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528354 2024-03-09T13:38:46.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528354 2024-03-09T13:38:46.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509507 3-313532612618 4-107374182460 5-360777252866 6-150323855405 7-171798691877 8-193273528354' 2024-03-09T13:38:46.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:38:46.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:38:47.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=335007449094 2024-03-09T13:38:47.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 335007449094 2024-03-09T13:38:47.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836564 1-42949673037 2-64424509507 3-313532612618 4-107374182460 5-360777252866 6-150323855405 7-171798691877 8-193273528354 9-335007449094' 2024-03-09T13:38:47.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:47.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836564 2024-03-09T13:38:47.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:47.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:38:47.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836564 2024-03-09T13:38:47.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:47.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836564 2024-03-09T13:38:47.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2024-03-09T13:38:47.069 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836564 2024-03-09T13:38:47.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:38:47.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836564 -lt 21474836564 2024-03-09T13:38:47.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:47.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673037 2024-03-09T13:38:47.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:47.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:38:47.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673037 2024-03-09T13:38:47.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:47.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673037 2024-03-09T13:38:47.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673037' 2024-03-09T13:38:47.349 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673037 2024-03-09T13:38:47.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:38:47.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673037 -lt 42949673037 2024-03-09T13:38:47.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:47.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509507 2024-03-09T13:38:47.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:47.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:38:47.625 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509507 2024-03-09T13:38:47.625 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:47.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509507 2024-03-09T13:38:47.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509507' 2024-03-09T13:38:47.627 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509507 2024-03-09T13:38:47.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:38:47.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509507 -lt 64424509507 2024-03-09T13:38:47.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:47.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612618 2024-03-09T13:38:47.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:47.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:38:47.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612618 2024-03-09T13:38:47.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:47.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612618 2024-03-09T13:38:47.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612618' 2024-03-09T13:38:47.903 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 313532612618 2024-03-09T13:38:47.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:38:48.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612618 -lt 313532612618 2024-03-09T13:38:48.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:48.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182460 2024-03-09T13:38:48.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:48.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:38:48.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182460 2024-03-09T13:38:48.180 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:48.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182460 2024-03-09T13:38:48.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182460' 2024-03-09T13:38:48.181 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182460 2024-03-09T13:38:48.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:38:48.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182460 -lt 107374182460 2024-03-09T13:38:48.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:48.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252866 2024-03-09T13:38:48.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:48.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:38:48.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252866 2024-03-09T13:38:48.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:48.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252866 2024-03-09T13:38:48.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252866' 2024-03-09T13:38:48.459 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 360777252866 2024-03-09T13:38:48.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:38:48.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252866 -lt 360777252866 2024-03-09T13:38:48.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:48.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855405 2024-03-09T13:38:48.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:48.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:38:48.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855405 2024-03-09T13:38:48.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:48.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855405 2024-03-09T13:38:48.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855405' 2024-03-09T13:38:48.734 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 150323855405 2024-03-09T13:38:48.734 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:38:49.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855406 -lt 150323855405 2024-03-09T13:38:49.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:49.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691877 2024-03-09T13:38:49.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:49.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:38:49.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691877 2024-03-09T13:38:49.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:49.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691877 2024-03-09T13:38:49.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691877' 2024-03-09T13:38:49.016 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691877 2024-03-09T13:38:49.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:38:49.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691877 -lt 171798691877 2024-03-09T13:38:49.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:49.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528354 2024-03-09T13:38:49.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:49.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:38:49.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528354 2024-03-09T13:38:49.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:49.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528354 2024-03-09T13:38:49.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528354' 2024-03-09T13:38:49.319 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528354 2024-03-09T13:38:49.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:38:49.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528354 -lt 193273528354 2024-03-09T13:38:49.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:38:49.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-335007449094 2024-03-09T13:38:49.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:38:49.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:38:49.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-335007449094 2024-03-09T13:38:49.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:38:49.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=335007449094 2024-03-09T13:38:49.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 335007449094' 2024-03-09T13:38:49.599 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 335007449094 2024-03-09T13:38:49.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:38:49.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 335007449094 -lt 335007449094 2024-03-09T13:38:49.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:38:49.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:38:49.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:38:50.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:38:50.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:38:50.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:38:50.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:38:50.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:38:50.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:38:50.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:38:50.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:38:50.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:38:50.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:38:50.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:38:50.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:38:50.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:38:50.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:38:50.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:38:50.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg ecpool SOMETHING 2024-03-09T13:38:50.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:38:50.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:38:50.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:38:50.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:38:51.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=2.0 2024-03-09T13:38:51.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 2.0 2024-03-09T13:38:51.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:38:51.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:38:51.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:38:51.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:38:51.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:38:51.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:38:51.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:38:22.877790+0000 2024-03-09T13:38:51.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:38:51.650 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:38:51.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:38:22.877790+0000 2024-03-09T13:38:51.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:38:51.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:38:22.877790+0000 2024-03-09T13:38:51.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:38:51.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:38:51.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:38:51.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:38:51.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:38:51.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:38:51.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:38:51.660 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:38:51.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:38:22.877790+0000 '>' 2024-03-09T13:38:22.877790+0000 2024-03-09T13:38:51.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:38:52.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:38:52.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:38:52.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:38:52.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:38:52.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:38:52.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:38:52.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:38:53.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:38:51.796520+0000 '>' 2024-03-09T13:38:22.877790+0000 2024-03-09T13:38:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:38:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:38:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:38:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:38:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T13:38:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:38:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:38:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:38:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:38:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T13:38:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:38:53.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T13:38:53.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:38:53.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:38:53.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:38:53.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:38:53.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:38:53.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:38:53.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:38:53.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:38:53.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:38:53.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T13:38:53.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:38:53.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:38:53.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-09T13:38:55.680 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:38:55.680 INFO:tasks.workunit.client.0.smithi162.stdout:hinfo_key 2024-03-09T13:38:55.680 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:38:55.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T13:38:55.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:38:55.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:38:55.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T13:38:55.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:38:55.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:38:55.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:38:55.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:38:55.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:38:55.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:38:55.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T13:38:55.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T13:38:55.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:38:55.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:38:55.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:38:55.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:38:55.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:38:55.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:38:55.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:38:55.953 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:38:55.954 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:38:55.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:38:55.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:38:55.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:38:55.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:38:55.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:38:55.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:38:55.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:38:55.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:38:55.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:38:55.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:38:55.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:38:55.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T13:38:55.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T13:38:55.955 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.9 2024-03-09T13:38:55.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:38:55.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T13:38:55.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T13:38:55.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:38:55.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:38:55.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:38:55.980 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:55.979+0000 7f65360c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:38:55.982 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:55.981+0000 7f65360c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:38:55.983 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:55.982+0000 7f65360c2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:38:56.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T13:38:56.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:38:56.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T13:38:56.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:38:56.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:38:56.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:56.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:38:56.242 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:38:56.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:56.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:38:56.498 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:56.497+0000 7f65360c2640 -1 Falling back to public interface 2024-03-09T13:38:56.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:38:57.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:38:57.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:57.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:38:57.519 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:38:57.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:57.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:38:57.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:38:58.765 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:38:58.763+0000 7f65360c2640 -1 osd.9 86 log_to_monitors true 2024-03-09T13:38:58.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:38:58.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:38:58.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:38:58.803 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:38:58.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:38:58.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:38:59.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:00.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:00.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:00.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:39:00.173 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:39:00.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:00.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:39:00.471 INFO:tasks.workunit.client.0.smithi162.stderr:osd.9 up in weight 1 up_from 91 up_thru 0 down_at 87 last_clean_interval [78,86) [v2:127.0.0.1:6874/3046862101,v1:127.0.0.1:6875/3046862101] [v2:127.0.0.1:6876/3046862101,v1:127.0.0.1:6877/3046862101] exists,up 0394dfb7-1912-4dc6-b9ea-633acdf67204 2024-03-09T13:39:00.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:39:00.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:39:00.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:39:00.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:39:00.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:39:00.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:39:00.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:39:00.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:39:00.472 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:39:00.472 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:39:00.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:39:00.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:39:00.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:39:00.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:39:00.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:39:00.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:39:00.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:39:00.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:39:00.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:39:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:39:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:39:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:39:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:39:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:4 2024-03-09T13:39:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:5 2024-03-09T13:39:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:6 2024-03-09T13:39:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:7 2024-03-09T13:39:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:8 2024-03-09T13:39:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:9' 2024-03-09T13:39:01.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:39:01.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:01.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:39:01.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836568 2024-03-09T13:39:01.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836568 2024-03-09T13:39:01.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568' 2024-03-09T13:39:01.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:01.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:39:01.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673041 2024-03-09T13:39:01.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673041 2024-03-09T13:39:01.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673041' 2024-03-09T13:39:01.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:01.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:39:01.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509511 2024-03-09T13:39:01.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509511 2024-03-09T13:39:01.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673041 2-64424509511' 2024-03-09T13:39:01.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:01.694 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:39:01.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612622 2024-03-09T13:39:01.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612622 2024-03-09T13:39:01.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673041 2-64424509511 3-313532612622' 2024-03-09T13:39:01.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:01.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:39:01.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182464 2024-03-09T13:39:01.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182464 2024-03-09T13:39:01.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673041 2-64424509511 3-313532612622 4-107374182464' 2024-03-09T13:39:01.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:01.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:39:02.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252870 2024-03-09T13:39:02.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252870 2024-03-09T13:39:02.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673041 2-64424509511 3-313532612622 4-107374182464 5-360777252870' 2024-03-09T13:39:02.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:02.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:39:02.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855410 2024-03-09T13:39:02.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855410 2024-03-09T13:39:02.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673041 2-64424509511 3-313532612622 4-107374182464 5-360777252870 6-150323855410' 2024-03-09T13:39:02.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:02.141 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:39:02.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691881 2024-03-09T13:39:02.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691881 2024-03-09T13:39:02.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673041 2-64424509511 3-313532612622 4-107374182464 5-360777252870 6-150323855410 7-171798691881' 2024-03-09T13:39:02.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:02.246 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:39:02.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528358 2024-03-09T13:39:02.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528358 2024-03-09T13:39:02.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673041 2-64424509511 3-313532612622 4-107374182464 5-360777252870 6-150323855410 7-171798691881 8-193273528358' 2024-03-09T13:39:02.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:02.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:39:02.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023938 2024-03-09T13:39:02.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023938 2024-03-09T13:39:02.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836568 1-42949673041 2-64424509511 3-313532612622 4-107374182464 5-360777252870 6-150323855410 7-171798691881 8-193273528358 9-390842023938' 2024-03-09T13:39:02.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:02.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836568 2024-03-09T13:39:02.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:02.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:39:02.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836568 2024-03-09T13:39:02.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:02.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836568 2024-03-09T13:39:02.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836568' 2024-03-09T13:39:02.458 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836568 2024-03-09T13:39:02.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:39:02.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836567 -lt 21474836568 2024-03-09T13:39:02.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:39:03.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:39:03.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:39:04.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836568 -lt 21474836568 2024-03-09T13:39:04.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:04.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673041 2024-03-09T13:39:04.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:04.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:39:04.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673041 2024-03-09T13:39:04.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:04.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673041 2024-03-09T13:39:04.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673041' 2024-03-09T13:39:04.019 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949673041 2024-03-09T13:39:04.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:39:04.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673041 -lt 42949673041 2024-03-09T13:39:04.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:04.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509511 2024-03-09T13:39:04.335 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:04.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:39:04.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509511 2024-03-09T13:39:04.336 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:04.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509511 2024-03-09T13:39:04.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509511' 2024-03-09T13:39:04.337 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509511 2024-03-09T13:39:04.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:39:04.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509511 -lt 64424509511 2024-03-09T13:39:04.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:04.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612622 2024-03-09T13:39:04.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:04.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:39:04.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612622 2024-03-09T13:39:04.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:04.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612622 2024-03-09T13:39:04.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612622' 2024-03-09T13:39:04.616 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.3 seq 313532612622 2024-03-09T13:39:04.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:39:04.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612623 -lt 313532612622 2024-03-09T13:39:04.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:04.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182464 2024-03-09T13:39:04.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:04.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:39:04.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182464 2024-03-09T13:39:04.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:04.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182464 2024-03-09T13:39:04.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182464' 2024-03-09T13:39:04.892 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.4 seq 107374182464 2024-03-09T13:39:04.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:39:05.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182464 -lt 107374182464 2024-03-09T13:39:05.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:05.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-360777252870 2024-03-09T13:39:05.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:05.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:39:05.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-360777252870 2024-03-09T13:39:05.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:05.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252870 2024-03-09T13:39:05.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 360777252870' 2024-03-09T13:39:05.171 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.5 seq 360777252870 2024-03-09T13:39:05.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:39:05.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252871 -lt 360777252870 2024-03-09T13:39:05.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:05.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855410 2024-03-09T13:39:05.447 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:05.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:39:05.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855410 2024-03-09T13:39:05.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:05.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855410 2024-03-09T13:39:05.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855410' 2024-03-09T13:39:05.449 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.6 seq 150323855410 2024-03-09T13:39:05.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:39:05.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855410 -lt 150323855410 2024-03-09T13:39:05.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:05.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691881 2024-03-09T13:39:05.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:05.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:39:05.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691881 2024-03-09T13:39:05.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:05.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691881 2024-03-09T13:39:05.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691881' 2024-03-09T13:39:05.742 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.7 seq 171798691881 2024-03-09T13:39:05.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:39:06.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691881 -lt 171798691881 2024-03-09T13:39:06.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:06.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528358 2024-03-09T13:39:06.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:06.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:39:06.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528358 2024-03-09T13:39:06.021 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:06.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528358 2024-03-09T13:39:06.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528358' 2024-03-09T13:39:06.022 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.8 seq 193273528358 2024-03-09T13:39:06.022 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:39:06.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528358 -lt 193273528358 2024-03-09T13:39:06.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:06.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-390842023938 2024-03-09T13:39:06.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:06.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:39:06.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-390842023938 2024-03-09T13:39:06.298 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:06.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023938 2024-03-09T13:39:06.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 390842023938' 2024-03-09T13:39:06.299 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.9 seq 390842023938 2024-03-09T13:39:06.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:39:06.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023939 -lt 390842023938 2024-03-09T13:39:06.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:39:06.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:06.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:06.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:39:06.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:39:06.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:39:06.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:39:06.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:39:06.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:39:06.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:39:06.915 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:39:07.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:39:07.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:39:07.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:07.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:07.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:39:07.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:39:07.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:39:07.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:39:07.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:39:07.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:262: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2024-03-09T13:39:07.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T13:39:07.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T13:39:07.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=5 2024-03-09T13:39:07.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=9 2024-03-09T13:39:07.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T13:39:07.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:39:07.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:39:07.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:39:07.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 269255"' 2024-03-09T13:39:07.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 269255' 2024-03-09T13:39:07.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T13:39:07.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:39:07.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:39:07.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 269256"' 2024-03-09T13:39:07.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 269256' 2024-03-09T13:39:07.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T13:39:07.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 269255 269256' 2024-03-09T13:39:07.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:39:07.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:39:07.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 269255 2024-03-09T13:39:07.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:39:07.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/269258: /' 2024-03-09T13:39:07.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T13:39:07.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/269260: /' 2024-03-09T13:39:10.783 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:39:10.783 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:39:10.783 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T13:39:10.783 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:39:10.783 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T13:39:10.783 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:39:10.783 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:39:10.783 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T13:39:10.783 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:39:10.783 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2024-03-09T13:39:10.784 INFO:tasks.workunit.client.0.smithi162.stderr:269260: remove 2#2:eb822e21:::SOMETHING:head# 2024-03-09T13:39:10.785 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T13:39:10.785 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:39:10.785 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:39:10.785 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T13:39:10.785 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:39:10.785 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:39:10.785 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:39:10.785 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:39:10.786 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:39:10.786 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:39:10.786 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T13:39:10.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T13:39:10.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:39:10.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:39:10.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:39:10.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:39:10.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:39:10.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:39:10.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:39:10.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T13:39:10.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T13:39:10.789 INFO:tasks.workunit.client.0.smithi162.stderr:269260: start osd.9 2024-03-09T13:39:10.789 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:39:10.822 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:39:10.822 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:39:10.822 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T13:39:10.822 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:39:10.822 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:39:10.822 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T13:39:10.823 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:39:10.824 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:39:10.824 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-09T13:39:10.824 INFO:tasks.workunit.client.0.smithi162.stderr:269258: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:39:10.824 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T13:39:10.824 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:39:10.824 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:39:10.824 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T13:39:10.824 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:39:10.824 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:39:10.824 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:39:10.824 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:39:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:39:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:39:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T13:39:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T13:39:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:39:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:39:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:39:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:39:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:39:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:39:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:39:10.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T13:39:10.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: start osd.5 2024-03-09T13:39:10.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:39:15.515 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T13:39:15.515 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T13:39:15.515 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:39:15.515 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:39:15.515 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:39:15.515 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 2024-03-09T13:39:10.814+0000 7fbaf8b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:15.515 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 2024-03-09T13:39:10.816+0000 7fbaf8b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:15.515 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 2024-03-09T13:39:10.817+0000 7fbaf8b2a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:15.515 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T13:39:15.515 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:39:15.515 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T13:39:15.515 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 0 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 2024-03-09T13:39:11.336+0000 7fbaf8b2a640 -1 Falling back to public interface 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 1 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:15.516 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 2 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 2024-03-09T13:39:14.074+0000 7fbaf8b2a640 -1 osd.9 93 log_to_monitors true 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 3 2024-03-09T13:39:15.517 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269260: osd.9 up in weight 1 up_from 99 up_thru 0 down_at 94 last_c269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 2024-03-09T13:39:10.851+0000 7f29074d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 2024-03-09T13:39:10.853+0000 7f29074d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 2024-03-09T13:39:10.854+0000 7f29074d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:39:15.518 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 0 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 2024-03-09T13:39:11.370+0000 7f29074d9640 -1 Falling back to public interface 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 1 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:15.519 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:39:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 2 2024-03-09T13:39:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:39:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 2024-03-09T13:39:14.169+0000 7f29074d9640 -1 osd.5 93 log_to_monitors true 2024-03-09T13:39:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:39:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 3 2024-03-09T13:39:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:39:16.387 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperlean_interval [91,93) [v2:127.0.0.1:6842/3095333061,v1:127.0.0.1:6843/3095333061] [v2:127.0.0.1:6844/3095333061,v1:127.0.0.1:6845/3095333061] exists,up 0394dfb7-1912-4dc6-b9ea-633acdf67204 2024-03-09T13:39:16.387 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:39:16.387 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:39:16.387 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:39:16.387 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:39:16.387 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:39:16.387 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:39:16.387 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:39:16.388 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 1 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 2 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 3 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 4 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 5 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 6 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 7 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 8 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: 9' 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836572 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836572 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572' 2024-03-09T13:39:16.389 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:16.390 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:39:16.390 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673045 2024-03-09T13:39:16.390 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673045 2024-03-09T13:39:16.390 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045' 2024-03-09T13:39:16.390 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:17.816 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qas.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:17.816 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:39:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 4 2024-03-09T13:39:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:39:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:269258: osd.5 up in weight 1 up_from 100 up_thru 100 down_at 94 last_clean_interval [84,93) [v2:127.0.0.1:6874/960654686,v1:127.0.0.1:6875/960654686] [v2:127.0.0.1:6876/960654686,v1:127.0.0.1:6877/960654686] exists,up 1a7a2a5c-3d7e-4a54-91c1-c81c5be3f535 2024-03-09T13:39:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:39:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:39:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:39:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:39:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:39:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:39:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 1 2024-03-09T13:39:17.818 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 2 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 3 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 4 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 5 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 6 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 7 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 8 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: 9' 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836573 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836573 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573' 2024-03-09T13:39:17.819 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:17.960 INFO:tasks.workunit.client.0.smithi162.stderr:269258: ///standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:39:17.960 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509515 2024-03-09T13:39:17.960 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509515 2024-03-09T13:39:17.960 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509515' 2024-03-09T13:39:17.960 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:17.960 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:39:17.960 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612626 2024-03-09T13:39:17.960 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612626 2024-03-09T13:39:17.960 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509515 3-313532612626' 2024-03-09T13:39:17.960 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:17.960 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182468 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182468 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509515 3-313532612626 4-107374182468' 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729602 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729602 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509515 3-313532612626 4-107374182468 5-429496729602' 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855414 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855414 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509515 3-313532612626 4-107374182468 5-429496729602 6-150323855414' 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:17.961 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:39:17.962 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691885 2024-03-09T13:39:17.962 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691885 2024-03-09T13:39:17.962 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509515 3-313532612626 4-107374182468 5-429496729602 6-150323855414 7-171798691885' 2024-03-09T13:39:17.962 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:17.962 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:39:18.639 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clonehome/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:39:18.639 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673046 2024-03-09T13:39:18.639 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673046 2024-03-09T13:39:18.639 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673046' 2024-03-09T13:39:18.639 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:18.639 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:39:18.639 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509516 2024-03-09T13:39:18.639 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509516 2024-03-09T13:39:18.639 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673046 2-64424509516' 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612628 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612628 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673046 2-64424509516 3-313532612628' 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182469 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182469 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673046 2-64424509516 3-313532612628 4-107374182469' 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729603 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729603 2024-03-09T13:39:18.640 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673046 2-64424509516 3-313532612628 4-107374182469 5-429496729603' 2024-03-09T13:39:18.641 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:18.641 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:39:18.641 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855415 2024-03-09T13:39:18.641 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855415 2024-03-09T13:39:18.641 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673046 2-64424509516 3-313532612628 4-107374182469 5-429496729603 6-150323855415' 2024-03-09T13:39:18.641 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:18.641 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:39:18.709 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_p.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528362 2024-03-09T13:39:18.709 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528362 2024-03-09T13:39:18.709 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509515 3-313532612626 4-107374182468 5-429496729602 6-150323855414 7-171798691885 8-193273528362' 2024-03-09T13:39:18.709 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:18.709 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:39:18.709 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-03-09T13:39:18.709 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-03-09T13:39:18.709 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836572 1-42949673045 2-64424509515 3-313532612626 4-107374182468 5-429496729602 6-150323855414 7-171798691885 8-193273528362 9-425201762306' 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836572 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836572 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836572 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: waiting osd.0 seq 21474836572 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836572 -lt 21474836572 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673045 2024-03-09T13:39:18.710 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:18.711 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:39:18.711 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673045 2024-03-09T13:39:18.711 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:18.711 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673045 2024-03-09T13:39:18.711 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673045' 2024-03-09T13:39:18.711 INFO:tasks.workunit.client.0.smithi162.stderr:269260: waiting osd.1 seq 42949673045 2024-03-09T13:39:18.711 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:39:18.711 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673045 -lt 42949673045 2024-03-09T13:39:18.711 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:18.711 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509515 2024-03-09T13:39:18.711 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:18.711 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:39:19.622 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509515 2024-03-09T13:39:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509515 2024-03-09T13:39:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509515' 2024-03-09T13:39:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:269260: waiting osd.2 seq 64424509515 2024-03-09T13:39:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:39:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509515 2024-03-09T13:39:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612626 2024-03-09T13:39:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:39:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612626 2024-03-09T13:39:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612626 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612626' 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: waiting osd.3 seq 313532612626 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612626 -lt 313532612626 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182468 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182468 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182468 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182468' 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: waiting osd.4 seq 107374182468 2024-03-09T13:39:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:39:19.625 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182470 -lt 107374182468 2024-03-09T13:39:19.625 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:19.625 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-429496729602 2024-03-09T13:39:19.625 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:19.625 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:39:19.625 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-429496729602 2024-03-09T13:39:19.625 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:19.625 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729602 2024-03-09T13:39:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waitig_stats: seq=171798691886 2024-03-09T13:39:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691886 2024-03-09T13:39:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673046 2-64424509516 3-313532612628 4-107374182469 5-429496729603 6-150323855415 7-171798691886' 2024-03-09T13:39:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:39:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528363 2024-03-09T13:39:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528363 2024-03-09T13:39:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673046 2-64424509516 3-313532612628 4-107374182469 5-429496729603 6-150323855415 7-171798691886 8-193273528363' 2024-03-09T13:39:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:39:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762307 2024-03-09T13:39:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762307 2024-03-09T13:39:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836573 1-42949673046 2-64424509516 3-313532612628 4-107374182469 5-429496729603 6-150323855415 7-171798691886 8-193273528363 9-425201762307' 2024-03-09T13:39:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836573 2024-03-09T13:39:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836573 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836573 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: waiting osd.0 seq 21474836573 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836572 -lt 21474836573 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836573 -lt 21474836573 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673046 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:39:20.453 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673046 2024-03-09T13:39:20.785 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ung osd.5 seq 429496729602' 2024-03-09T13:39:20.786 INFO:tasks.workunit.client.0.smithi162.stderr:269260: waiting osd.5 seq 429496729602 2024-03-09T13:39:20.786 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:39:20.786 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729603 -lt 429496729602 2024-03-09T13:39:20.786 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:20.786 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855414 2024-03-09T13:39:20.786 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:20.786 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:39:20.786 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855414 2024-03-09T13:39:20.786 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:20.786 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855414 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855414' 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: waiting osd.6 seq 150323855414 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855415 -lt 150323855414 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691885 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691885 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691885 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691885' 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: waiting osd.7 seq 171798691885 2024-03-09T13:39:20.787 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:39:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691886 -lt 171798691885 2024-03-09T13:39:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528362 2024-03-09T13:39:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:39:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528362 2024-03-09T13:39:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528362 2024-03-09T13:39:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528362' 2024-03-09T13:39:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: waiting osd.8 seq 193273528362 2024-03-09T13:39:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:39:20.788 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528363 -lt 193273528362 2024-03-09T13:39:21.423 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pgbuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:21.423 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673046 2024-03-09T13:39:21.423 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: waiting osd.1 seq 42949673046 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673047 -lt 42949673046 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509516 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509516 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509516 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509516' 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: waiting osd.2 seq 64424509516 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:39:21.424 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509516 -lt 64424509516 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612628 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612628 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612628 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612628' 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: waiting osd.3 seq 313532612628 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612628 -lt 313532612628 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182469 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:21.425 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:39:21.426 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182469 2024-03-09T13:39:21.426 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:21.426 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182469 2024-03-09T13:39:21.426 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182469' 2024-03-09T13:39:21.426 INFO:tasks.workunit.client.0.smithi162.stderr:269258: waiting osd.4 seq 107374182469 2024-03-09T13:39:22.154 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone._stats: for s in $seqs 2024-03-09T13:39:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-425201762306 2024-03-09T13:39:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:39:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-425201762306 2024-03-09T13:39:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-03-09T13:39:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 425201762306' 2024-03-09T13:39:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:269260: waiting osd.9 seq 425201762306 2024-03-09T13:39:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:39:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762307 -lt 425201762306 2024-03-09T13:39:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:39:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:39:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:39:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:39:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:39:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:39:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:39:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:39:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:39:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:39:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:39:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:269260: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:39:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:39:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:269260: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:39:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:39:22.654 INFO:tasks.workunit.client.0.smithi162.stderr:client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:39:22.654 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182470 -lt 107374182469 2024-03-09T13:39:22.654 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:22.654 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-429496729603 2024-03-09T13:39:22.654 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:22.654 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:39:22.654 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-429496729603 2024-03-09T13:39:22.654 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:22.654 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729603 2024-03-09T13:39:22.654 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 429496729603' 2024-03-09T13:39:22.655 INFO:tasks.workunit.client.0.smithi162.stderr:269258: waiting osd.5 seq 429496729603 2024-03-09T13:39:22.655 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:39:22.655 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729604 -lt 429496729603 2024-03-09T13:39:22.655 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:22.655 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855415 2024-03-09T13:39:22.655 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:22.655 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:39:22.655 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855415 2024-03-09T13:39:22.655 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:22.655 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855415 2024-03-09T13:39:22.655 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855415' 2024-03-09T13:39:22.655 INFO:tasks.workunit.client.0.smithi162.stderr:269258: waiting osd.6 seq 150323855415 2024-03-09T13:39:22.655 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:39:22.656 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855415 -lt 150323855415 2024-03-09T13:39:22.656 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:22.656 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691886 2024-03-09T13:39:22.656 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:22.656 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:39:22.656 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691886 2024-03-09T13:39:22.656 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:22.656 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691886 2024-03-09T13:39:22.656 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691886' 2024-03-09T13:39:22.656 INFO:tasks.workunit.client.0.smithi162.stderr:269258: waiting osd.7 seq 171798691886 2024-03-09T13:39:22.656 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:39:22.656 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691887 -lt 171798691886 2024-03-09T13:39:22.656 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:23.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528363 2024-03-09T13:39:23.825 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528363 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528363 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528363' 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: waiting osd.8 seq 193273528363 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528363 -lt 193273528363 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-425201762307 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-425201762307 2024-03-09T13:39:23.826 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762307 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 425201762307' 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: waiting osd.9 seq 425201762307 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762308 -lt 425201762307 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:39:23.827 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:39:23.828 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:39:23.828 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:39:23.828 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:39:23.828 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:39:24.169 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:24.169 INFO:tasks.workunit.client.0.smithi162.stderr:269258: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:24.169 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:39:24.169 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:39:24.169 INFO:tasks.workunit.client.0.smithi162.stderr:269258: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:39:24.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:39:24.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:39:24.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 269256 2024-03-09T13:39:24.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:39:24.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:39:24.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:39:24.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T13:39:24.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:39:24.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T13:39:24.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:39:24.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:39:24.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:39:24.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:39:24.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T13:39:24.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T13:39:24.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:39:24.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:39:24.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:39:24.448 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:39:24.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:39:24.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:39:24.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:38:51.796520+0000 2024-03-09T13:39:24.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:39:24.981 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:39:24.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:38:51.796520+0000 2024-03-09T13:39:24.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:39:24.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:38:51.796520+0000 2024-03-09T13:39:24.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:39:24.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:39:24.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:39:24.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:39:24.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:39:24.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:39:24.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:39:24.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:39:25.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:38:51.796520+0000 '>' 2024-03-09T13:38:51.796520+0000 2024-03-09T13:39:25.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:39:26.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:39:26.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:39:26.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:39:26.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:39:26.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:39:26.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:39:26.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:39:26.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:38:51.796520+0000 '>' 2024-03-09T13:38:51.796520+0000 2024-03-09T13:39:26.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:39:27.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:39:27.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:39:27.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:39:27.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:39:27.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:39:27.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:39:27.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:39:27.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:38:51.796520+0000 '>' 2024-03-09T13:38:51.796520+0000 2024-03-09T13:39:27.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:39:28.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:39:28.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:39:28.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:39:28.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:39:28.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:39:28.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:39:28.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:39:29.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:38:51.796520+0000 '>' 2024-03-09T13:38:51.796520+0000 2024-03-09T13:39:29.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:39:30.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:39:30.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:39:30.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:39:30.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:39:30.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:39:30.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:39:30.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:39:30.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:39:25.537430+0000 '>' 2024-03-09T13:38:51.796520+0000 2024-03-09T13:39:30.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:39:30.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T13:39:30.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:39:30.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:39:30.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:39:30.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 272708"' 2024-03-09T13:39:30.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 272708' 2024-03-09T13:39:30.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:39:30.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:39:30.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:39:30.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 272710"' 2024-03-09T13:39:30.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 272710' 2024-03-09T13:39:30.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:39:30.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T13:39:30.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 272708 272710' 2024-03-09T13:39:30.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:39:30.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:39:30.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 272708 2024-03-09T13:39:30.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/272711: /' 2024-03-09T13:39:30.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:39:30.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/272713: /' 2024-03-09T13:39:33.276 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:39:33.276 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:39:33.276 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-03-09T13:39:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:39:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:39:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:39:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:39:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-03-09T13:39:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:39:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2024-03-09T13:39:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:39:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:39:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:39:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:39:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:39:33.277 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: _ 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: hinfo_key 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: snapset 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:39:33.278 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-03-09T13:39:33.279 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:39:33.279 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/9 2024-03-09T13:39:33.279 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:39:33.279 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:39:33.279 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:39:33.279 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:39:33.279 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2024-03-09T13:39:33.279 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2024-03-09T13:39:33.279 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:39:33.280 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:39:33.281 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:39:33.281 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:39:33.281 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:39:33.281 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:39:33.282 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/9 2024-03-09T13:39:33.282 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-03-09T13:39:33.282 INFO:tasks.workunit.client.0.smithi162.stderr:272713: start osd.9 2024-03-09T13:39:33.367 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_rec272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:39:33.368 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:39:33.368 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T13:39:33.368 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:39:33.368 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:39:33.368 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:39:33.368 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:39:33.368 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T13:39:33.368 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:39:33.368 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T13:39:33.368 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:39:33.368 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:39:33.368 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: _ 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: hinfo_key 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: snapset 2024-03-09T13:39:33.369 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T13:39:33.370 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:39:33.370 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:39:33.370 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T13:39:33.370 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:39:33.370 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:39:33.370 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:39:33.370 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:39:33.370 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:39:33.371 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:39:33.371 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T13:39:33.371 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T13:39:33.371 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:39:33.371 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:39:33.371 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:39:33.371 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:39:33.371 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:39:33.371 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:39:33.371 INFO:tasks.workunit.client.0.smithi162.stderr:272711: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:39:33.371 INFO:tasks.workunit.client.0.smithi162.stderr:272711: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:39:33.371 INFO:tasks.workunit.client.0.smithi162.stderr:272711: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T13:39:33.372 INFO:tasks.workunit.client.0.smithi162.stderr:272711: start osd.5 2024-03-09T13:39:37.967 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:39:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/9/whoami 2024-03-09T13:39:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-03-09T13:39:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:39:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:39:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:39:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 2024-03-09T13:39:33.307+0000 7f5c814c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 2024-03-09T13:39:33.309+0000 7f5c814c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 2024-03-09T13:39:33.310+0000 7f5c814c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-03-09T13:39:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:39:37.968 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 0 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 2024-03-09T13:39:33.828+0000 7f5c814c6640 -1 Falling back to public interface 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 1 2024-03-09T13:39:37.969 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:39:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:39:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 2 2024-03-09T13:39:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:39:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 2024-03-09T13:39:36.410+0000 7f5c814c6640 -1 osd.9 103 log_to_monitors true 2024-03-09T13:39:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:37.970 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:39:37.971 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 3 2024-03-09T13:39:37.971 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:37.971 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-03-09T13:39:38.168 INFO:tasks.workunit.client.0.smithi162.stderr:272713: osd.9 up in weight 1 up_from 107 up_thru 0 down_overy_ops 2024-03-09T13:39:38.168 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T13:39:38.168 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T13:39:38.168 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:39:38.168 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:39:38.168 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:39:38.168 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 2024-03-09T13:39:33.406+0000 7f083eeb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:38.168 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 2024-03-09T13:39:33.434+0000 7f083eeb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:38.168 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 2024-03-09T13:39:33.435+0000 7f083eeb1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:38.168 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T13:39:38.169 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:39:38.169 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T13:39:38.169 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:39:38.169 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:39:38.169 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:38.169 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:39:38.169 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 0 2024-03-09T13:39:38.169 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:38.169 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:39:38.169 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 2024-03-09T13:39:33.954+0000 7f083eeb1640 -1 Falling back to public interface 2024-03-09T13:39:38.170 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:38.170 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:38.170 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:38.170 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:39:38.170 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 1 2024-03-09T13:39:38.170 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:38.170 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:39:38.170 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:38.170 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:38.170 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:38.171 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:39:38.171 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 2 2024-03-09T13:39:38.171 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:38.171 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:39:38.171 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:38.171 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 2024-03-09T13:39:36.850+0000 7f083eeb1640 -1 osd.5 102 log_to_monitors true 2024-03-09T13:39:38.171 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:38.171 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:38.171 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:39:38.171 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 3 2024-03-09T13:39:38.171 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:38.171 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:39:38.763 INFO:tasks.workunit.client.0.smithi162.stderr:272711: osd.5 up in weight 1 up_from 107 up_thru 100 dowat 103 last_clean_interval [99,103) [v2:127.0.0.1:6842/2120340746,v1:127.0.0.1:6843/2120340746] [v2:127.0.0.1:6844/2120340746,v1:127.0.0.1:6845/2120340746] exists,up 0394dfb7-1912-4dc6-b9ea-633acdf67204 2024-03-09T13:39:38.764 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:39:38.764 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:39:38.764 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:39:38.764 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:39:38.764 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:39:38.764 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:39:38.764 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:39:38.764 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:39:38.764 INFO:tasks.workunit.client.0.smithi162.stderr:272713: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:39:38.764 INFO:tasks.workunit.client.0.smithi162.stderr:272713: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:39:38.764 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:39:38.764 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:39:38.764 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 1 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 2 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 3 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 4 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 5 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 6 2024-03-09T13:39:38.765 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 7 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 8 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: 9' 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836578 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836578 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578' 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673051 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673051 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051' 2024-03-09T13:39:38.766 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clon_at 103 last_clean_interval [100,102) [v2:127.0.0.1:6874/2240959128,v1:127.0.0.1:6875/2240959128] [v2:127.0.0.1:6876/2240959128,v1:127.0.0.1:6877/2240959128] exists,up 1a7a2a5c-3d7e-4a54-91c1-c81c5be3f535 2024-03-09T13:39:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:39:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:39:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:39:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:39:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:39:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:39:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:39:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 1 2024-03-09T13:39:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 2 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 3 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 4 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 5 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 6 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 7 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 8 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: 9' 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836579 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836579 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579' 2024-03-09T13:39:39.073 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:39.074 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:39:39.074 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673053 2024-03-09T13:39:39.074 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673053 2024-03-09T13:39:39.074 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673053' 2024-03-09T13:39:39.074 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.152 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/ne.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:39:40.152 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509521 2024-03-09T13:39:40.152 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509521 2024-03-09T13:39:40.152 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509521' 2024-03-09T13:39:40.152 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.152 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612633 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612633 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509521 3-313532612633' 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182475 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182475 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509521 3-313532612633 4-107374182475' 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500674 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500674 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509521 3-313532612633 4-107374182475 5-459561500674' 2024-03-09T13:39:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.154 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:39:40.154 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855420 2024-03-09T13:39:40.154 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855420 2024-03-09T13:39:40.154 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509521 3-313532612633 4-107374182475 5-459561500674 6-150323855420' 2024-03-09T13:39:40.154 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.154 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:39:40.154 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691891 2024-03-09T13:39:40.154 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691891 2024-03-09T13:39:40.154 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509521 3-313532612633 4-107374182475 5-459561500674 6-150323855420 7-171798691891' 2024-03-09T13:39:40.154 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.154 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:39:40.348 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509522 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509522 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673053 2-64424509522' 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612634 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612634 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673053 2-64424509522 3-313532612634' 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182476 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182476 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673053 2-64424509522 3-313532612634 4-107374182476' 2024-03-09T13:39:40.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500675 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500675 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673053 2-64424509522 3-313532612634 4-107374182476 5-459561500675' 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855421 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855421 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673053 2-64424509522 3-313532612634 4-107374182476 5-459561500675 6-150323855421' 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691892 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691892 2024-03-09T13:39:40.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673053 2-64424509522 3-313532612634 4-107374182476 5-459561500675 6-150323855421 7-171798691892' 2024-03-09T13:39:40.351 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.351 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:39:40.933 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuncephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528368 2024-03-09T13:39:40.933 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528368 2024-03-09T13:39:40.933 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509521 3-313532612633 4-107374182475 5-459561500674 6-150323855420 7-171798691891 8-193273528368' 2024-03-09T13:39:40.933 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:40.933 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:39:40.933 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500674 2024-03-09T13:39:40.933 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500674 2024-03-09T13:39:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836578 1-42949673051 2-64424509521 3-313532612633 4-107374182475 5-459561500674 6-150323855420 7-171798691891 8-193273528368 9-459561500674' 2024-03-09T13:39:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836578 2024-03-09T13:39:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:39:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836578 2024-03-09T13:39:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836578 2024-03-09T13:39:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-03-09T13:39:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:272713: waiting osd.0 seq 21474836578 2024-03-09T13:39:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:39:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836578 2024-03-09T13:39:40.934 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673051 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673051 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673051 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673051' 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: waiting osd.1 seq 42949673051 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673053 -lt 42949673051 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509521 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:40.935 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:39:41.100 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cetu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-03-09T13:39:41.100 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-03-09T13:39:41.101 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673053 2-64424509522 3-313532612634 4-107374182476 5-459561500675 6-150323855421 7-171798691892 8-193273528369' 2024-03-09T13:39:41.101 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:41.101 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:39:41.101 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500675 2024-03-09T13:39:41.101 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500675 2024-03-09T13:39:41.101 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836579 1-42949673053 2-64424509522 3-313532612634 4-107374182476 5-459561500675 6-150323855421 7-171798691892 8-193273528369 9-459561500675' 2024-03-09T13:39:41.101 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:41.101 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836579 2024-03-09T13:39:41.101 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:41.101 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:39:41.101 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836579 2024-03-09T13:39:41.101 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836579 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: waiting osd.0 seq 21474836579 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836579 -lt 21474836579 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673053 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673053 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673053 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673053' 2024-03-09T13:39:41.102 INFO:tasks.workunit.client.0.smithi162.stderr:272711: waiting osd.1 seq 42949673053 2024-03-09T13:39:41.103 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:39:41.103 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673053 -lt 42949673053 2024-03-09T13:39:41.103 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:41.103 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509522 2024-03-09T13:39:41.103 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:41.103 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:39:41.858 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntuphtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509521 2024-03-09T13:39:41.858 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:41.858 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509521 2024-03-09T13:39:41.858 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-03-09T13:39:41.859 INFO:tasks.workunit.client.0.smithi162.stderr:272713: waiting osd.2 seq 64424509521 2024-03-09T13:39:41.859 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:39:41.859 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509522 -lt 64424509521 2024-03-09T13:39:41.859 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:41.859 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612633 2024-03-09T13:39:41.859 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:41.859 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:39:41.859 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612633 2024-03-09T13:39:41.859 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612633 2024-03-09T13:39:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612633' 2024-03-09T13:39:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:272713: waiting osd.3 seq 313532612633 2024-03-09T13:39:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:39:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612634 -lt 313532612633 2024-03-09T13:39:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182475 2024-03-09T13:39:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:39:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182475 2024-03-09T13:39:41.860 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182475 2024-03-09T13:39:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182475' 2024-03-09T13:39:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:272713: waiting osd.4 seq 107374182475 2024-03-09T13:39:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:39:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182476 -lt 107374182475 2024-03-09T13:39:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-459561500674 2024-03-09T13:39:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:39:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-459561500674 2024-03-09T13:39:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:41.861 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500674 2024-03-09T13:39:42.085 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509522 2024-03-09T13:39:42.086 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:42.086 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509522 2024-03-09T13:39:42.086 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509522' 2024-03-09T13:39:42.086 INFO:tasks.workunit.client.0.smithi162.stderr:272711: waiting osd.2 seq 64424509522 2024-03-09T13:39:42.086 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:39:42.086 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509523 -lt 64424509522 2024-03-09T13:39:42.086 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:42.086 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-313532612634 2024-03-09T13:39:42.086 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:42.086 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:39:42.086 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-313532612634 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612634 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 313532612634' 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: waiting osd.3 seq 313532612634 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612634 -lt 313532612634 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182476 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182476 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182476 2024-03-09T13:39:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182476' 2024-03-09T13:39:42.088 INFO:tasks.workunit.client.0.smithi162.stderr:272711: waiting osd.4 seq 107374182476 2024-03-09T13:39:42.088 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:39:42.088 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182476 -lt 107374182476 2024-03-09T13:39:42.088 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:42.088 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-459561500675 2024-03-09T13:39:42.088 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:42.088 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:39:42.088 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-459561500675 2024-03-09T13:39:42.088 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:42.088 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500675 2024-03-09T13:39:43.082 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_st: echo 'waiting osd.5 seq 459561500674' 2024-03-09T13:39:43.082 INFO:tasks.workunit.client.0.smithi162.stderr:272713: waiting osd.5 seq 459561500674 2024-03-09T13:39:43.082 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:39:43.082 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500675 -lt 459561500674 2024-03-09T13:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855420 2024-03-09T13:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855420 2024-03-09T13:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855420 2024-03-09T13:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855420' 2024-03-09T13:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:272713: waiting osd.6 seq 150323855420 2024-03-09T13:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855421 -lt 150323855420 2024-03-09T13:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:43.083 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691891 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691891 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691891 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691891' 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: waiting osd.7 seq 171798691891 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691893 -lt 171798691891 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528368 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:39:43.084 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528368 2024-03-09T13:39:43.085 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:43.085 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528368 2024-03-09T13:39:43.085 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528368' 2024-03-09T13:39:43.085 INFO:tasks.workunit.client.0.smithi162.stderr:272713: waiting osd.8 seq 193273528368 2024-03-09T13:39:43.085 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:39:43.085 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528369 -lt 193273528368 2024-03-09T13:39:43.347 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:ats: echo 'waiting osd.5 seq 459561500675' 2024-03-09T13:39:43.347 INFO:tasks.workunit.client.0.smithi162.stderr:272711: waiting osd.5 seq 459561500675 2024-03-09T13:39:43.347 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:39:43.347 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500675 -lt 459561500675 2024-03-09T13:39:43.347 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:43.347 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855421 2024-03-09T13:39:43.347 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:43.348 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:39:43.348 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855421 2024-03-09T13:39:43.348 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:43.348 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855421 2024-03-09T13:39:43.348 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855421' 2024-03-09T13:39:43.348 INFO:tasks.workunit.client.0.smithi162.stderr:272711: waiting osd.6 seq 150323855421 2024-03-09T13:39:43.348 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:39:43.348 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855421 -lt 150323855421 2024-03-09T13:39:43.348 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:43.348 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691892 2024-03-09T13:39:43.348 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:43.348 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:39:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691892 2024-03-09T13:39:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691892 2024-03-09T13:39:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691892' 2024-03-09T13:39:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: waiting osd.7 seq 171798691892 2024-03-09T13:39:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:39:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691893 -lt 171798691892 2024-03-09T13:39:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-03-09T13:39:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:39:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-03-09T13:39:43.349 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:43.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-03-09T13:39:43.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-03-09T13:39:43.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: waiting osd.8 seq 193273528369 2024-03-09T13:39:43.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:39:43.350 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528369 2024-03-09T13:39:44.533 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:44.533 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500674 2024-03-09T13:39:44.533 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:44.533 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:39:44.533 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500674 2024-03-09T13:39:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500674 2024-03-09T13:39:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500674' 2024-03-09T13:39:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:272713: waiting osd.9 seq 459561500674 2024-03-09T13:39:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:39:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500676 -lt 459561500674 2024-03-09T13:39:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:39:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:39:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:39:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:39:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:39:44.535 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:39:44.536 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:39:44.536 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:39:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:2727sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500675 2024-03-09T13:39:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:39:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500675 2024-03-09T13:39:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500675 2024-03-09T13:39:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500675' 2024-03-09T13:39:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:272711: waiting osd.9 seq 459561500675 2024-03-09T13:39:44.765 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500676 -lt 459561500675 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:39:44.766 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:39:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T13:39:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T13:39:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:39:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T13:39:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:39:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:39:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:39:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:39:44.767 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:39:45.753 INFO:tasks.workunit.client.0.smithi162.stderr:213: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:39:45.753 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:39:45.753 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:39:45.753 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:39:45.753 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:39:45.753 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:39:45.753 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:39:45.753 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:39:45.754 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:39:45.755 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:39:45.755 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:39:45.755 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:39:45.755 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:39:45.755 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:39:45.755 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:39:45.755 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:39:45.950 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format72711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:39:45.950 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:39:45.950 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:39:45.950 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:39:45.951 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:39:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:39:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:39:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:39:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:39:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:39:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:39:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:39:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:39:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:39:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:39:45.952 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:39:46.386 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --for json pg dump pgs 2024-03-09T13:39:46.386 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:39:46.386 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:39:46.386 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:39:46.386 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:46.386 INFO:tasks.workunit.client.0.smithi162.stderr:272713: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:46.386 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:39:46.386 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:39:46.387 INFO:tasks.workunit.client.0.smithi162.stderr:272713: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:39:46.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:39:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:mat json pg dump pgs 2024-03-09T13:39:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:39:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:39:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:39:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:39:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:272711: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:39:46.581 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:39:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:39:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:272711: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:39:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:39:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:39:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 272710 2024-03-09T13:39:46.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:39:46.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:39:46.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:39:46.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T13:39:46.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:39:46.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:39:46.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:39:46.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:185: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:186: corrupt_and_repair_two: local poolname=ecpool 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: corrupt_and_repair_two: local first=3 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: corrupt_and_repair_two: local second=5 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:193: corrupt_and_repair_two: pids= 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 276185"' 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 276185' 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:39:46.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 276187"' 2024-03-09T13:39:46.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:39:46.623 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 276187' 2024-03-09T13:39:46.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: corrupt_and_repair_two: wait_background pids 2024-03-09T13:39:46.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 276185 276187' 2024-03-09T13:39:46.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:39:46.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:39:46.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 276185 2024-03-09T13:39:46.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/276188: /' 2024-03-09T13:39:46.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:39:46.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/276190: /' 2024-03-09T13:39:50.003 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:39:50.004 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:39:50.004 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T13:39:50.004 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:39:50.004 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:39:50.004 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:39:50.004 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:39:50.004 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T13:39:50.004 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:39:50.004 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T13:39:50.004 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: remove 1#2:eb822e21:::SOMETHING:head# 2024-03-09T13:39:50.005 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T13:39:50.006 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:39:50.006 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:39:50.006 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T13:39:50.006 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:39:50.006 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:39:50.006 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:39:50.006 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:39:50.007 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:39:50.007 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:39:50.007 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T13:39:50.007 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T13:39:50.007 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:39:50.007 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:39:50.007 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:39:50.007 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:39:50.007 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:39:50.008 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:39:50.009 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:39:50.009 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T13:39:50.009 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T13:39:50.009 INFO:tasks.workunit.client.0.smithi162.stderr:276190: start osd.5 2024-03-09T13:39:50.009 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:39:50.183 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:39:50.183 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:39:50.183 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:39:50.183 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:39:50.183 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:39:50.183 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:39:50.183 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:39:50.183 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:39:50.184 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:39:50.184 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:39:50.184 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:39:50.184 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:39:50.184 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:39:50.184 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:39:50.184 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:39:50.184 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:39:50.184 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2024-03-09T13:39:50.184 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:39:50.184 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:39:50.184 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:39:50.184 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:39:50.185 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:39:50.185 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2024-03-09T13:39:50.185 INFO:tasks.workunit.client.0.smithi162.stderr:276188: remove 0#2:eb822e21:::SOMETHING:head# 2024-03-09T13:39:50.185 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:39:50.185 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:39:50.185 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:39:50.185 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:39:50.185 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:39:50.185 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:39:50.185 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:39:50.185 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:39:50.186 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:39:50.186 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:39:50.186 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:39:50.186 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:39:50.186 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:39:50.186 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:39:50.186 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:39:50.186 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:39:50.186 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:39:50.186 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:39:50.186 INFO:tasks.workunit.client.0.smithi162.stderr:276188: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:39:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:276188: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:39:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:276188: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:39:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:39:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:39:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:39:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:39:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:39:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:39:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:39:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:39:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:39:50.187 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:39:50.188 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:39:50.188 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:39:50.189 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:39:50.189 INFO:tasks.workunit.client.0.smithi162.stderr:276188: start osd.3 2024-03-09T13:39:50.189 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:39:54.569 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T13:39:54.569 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T13:39:54.569 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:39:54.570 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:39:54.570 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:39:54.570 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 2024-03-09T13:39:50.034+0000 7f2cc7338640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:54.570 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 2024-03-09T13:39:50.037+0000 7f2cc7338640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:54.570 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 2024-03-09T13:39:50.038+0000 7f2cc7338640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:54.570 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T13:39:54.570 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:39:54.570 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T13:39:54.570 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:39:54.570 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:39:54.571 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:54.571 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:39:54.571 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 0 2024-03-09T13:39:54.571 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:54.571 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:39:54.571 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 2024-03-09T13:39:50.564+0000 7f2cc7338640 -1 Falling back to public interface 2024-03-09T13:39:54.571 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:54.571 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:54.571 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:54.571 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:39:54.571 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 1 2024-03-09T13:39:54.571 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:54.571 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:39:54.572 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:54.572 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:54.572 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:54.572 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:39:54.572 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 2 2024-03-09T13:39:54.572 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:54.572 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:39:54.572 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:54.572 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 2024-03-09T13:39:53.249+0000 7f2cc7338640 -1 osd.5 110 log_to_monitors true 2024-03-09T13:39:54.572 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:54.572 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:54.573 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:39:54.573 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 3 2024-03-09T13:39:54.573 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:54.573 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:39:54.850 INFO:tasks.workunit.client.0.smithi162.stderr:276190: osd.5 up in weight 1 up_from 114 up_thru 114 down_at 111 l276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:39:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:39:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:39:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:39:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:39:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 2024-03-09T13:39:50.210+0000 7fb9d14f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 2024-03-09T13:39:50.212+0000 7fb9d14f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 2024-03-09T13:39:50.213+0000 7fb9d14f9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:39:54.851 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 0 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 2024-03-09T13:39:50.734+0000 7fb9d14f9640 -1 Falling back to public interface 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:54.852 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 1 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 2 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 2024-03-09T13:39:53.535+0000 7fb9d14f9640 -1 osd.3 110 log_to_monitors true 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:39:54.853 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:39:54.854 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 3 2024-03-09T13:39:54.854 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:39:54.854 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:39:55.428 INFO:tasks.workunit.client.0.smithi162.stderr:276188: osd.3 up in weight 1 up_from 115 up_thru 107 down_at 111 last_clean_interval [107,110) [v2:127.0.0.1:6826/1060398780,v1:127.0.0.1:6827/1060398780] [v2:127.0.0.1:6828/1060398780,v1:127.0.0.1:6829/1060398780] exists,up 1a7a2a5c-3d7e-4a54-91c1-c81c5be3f535 2024-03-09T13:39:55.428 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:39:55.429 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:39:55.429 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:39:55.429 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:39:55.429 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:39:55.429 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:39:55.429 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:39:55.430 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:39:55.430 INFO:tasks.workunit.client.0.smithi162.stderr:276190: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:39:55.430 INFO:tasks.workunit.client.0.smithi162.stderr:276190: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:39:55.430 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:39:55.430 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:39:55.430 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:39:55.431 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:39:55.431 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:39:55.431 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:39:55.431 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:39:55.431 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:39:55.431 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:39:55.432 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:39:55.432 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 1 2024-03-09T13:39:55.432 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 2 2024-03-09T13:39:55.432 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 3 2024-03-09T13:39:55.432 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 4 2024-03-09T13:39:55.433 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 5 2024-03-09T13:39:55.433 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 6 2024-03-09T13:39:55.433 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 7 2024-03-09T13:39:55.433 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 8 2024-03-09T13:39:55.433 INFO:tasks.workunit.client.0.smithi162.stderr:276190: 9' 2024-03-09T13:39:55.433 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:39:55.433 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:55.434 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:39:55.434 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836584 2024-03-09T13:39:55.434 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836584 2024-03-09T13:39:55.434 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584' 2024-03-09T13:39:55.434 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:55.435 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:39:55.435 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673057 2024-03-09T13:39:55.435 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673057 2024-03-09T13:39:55.435 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057' 2024-03-09T13:39:55.435 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:55.757 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.clieast_clean_interval [73,110) [v2:127.0.0.1:6874/4126112885,v1:127.0.0.1:6875/4126112885] [v2:127.0.0.1:6876/4126112885,v1:127.0.0.1:6877/4126112885] exists,up 2095e0ae-504c-4658-8991-61158f227389 2024-03-09T13:39:55.757 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:39:55.757 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:39:55.757 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:39:55.757 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:39:55.757 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:39:55.757 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:39:55.757 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:39:55.758 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 1 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 2 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 3 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 4 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 5 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 6 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 7 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 8 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: 9' 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836585 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836585 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585' 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:55.759 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:39:55.760 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673058 2024-03-09T13:39:55.760 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673058 2024-03-09T13:39:55.760 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673058' 2024-03-09T13:39:55.760 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:56.453 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.cliennt.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:39:56.453 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509526 2024-03-09T13:39:56.454 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509526 2024-03-09T13:39:56.454 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509526' 2024-03-09T13:39:56.454 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:56.454 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:39:56.454 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239042 2024-03-09T13:39:56.454 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239042 2024-03-09T13:39:56.455 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509526 3-493921239042' 2024-03-09T13:39:56.455 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:56.455 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:39:56.455 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182480 2024-03-09T13:39:56.455 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182480 2024-03-09T13:39:56.455 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509526 3-493921239042 4-107374182480' 2024-03-09T13:39:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:39:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271746 2024-03-09T13:39:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271746 2024-03-09T13:39:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509526 3-493921239042 4-107374182480 5-489626271746' 2024-03-09T13:39:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:39:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855425 2024-03-09T13:39:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855425 2024-03-09T13:39:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509526 3-493921239042 4-107374182480 5-489626271746 6-150323855425' 2024-03-09T13:39:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:39:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691897 2024-03-09T13:39:56.457 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691897 2024-03-09T13:39:56.457 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509526 3-493921239042 4-107374182480 5-489626271746 6-150323855425 7-171798691897' 2024-03-09T13:39:56.457 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:56.457 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:39:56.561 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:39:56.561 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509527 2024-03-09T13:39:56.562 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509527 2024-03-09T13:39:56.562 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509527' 2024-03-09T13:39:56.562 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:56.562 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:39:56.562 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=493921239043 2024-03-09T13:39:56.562 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 493921239043 2024-03-09T13:39:56.562 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509527 3-493921239043' 2024-03-09T13:39:56.562 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:56.562 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:39:56.562 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182481 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182481 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509527 3-493921239043 4-107374182481' 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=489626271747 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 489626271747 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509527 3-493921239043 4-107374182481 5-489626271747' 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855426 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855426 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509527 3-493921239043 4-107374182481 5-489626271747 6-150323855426' 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:39:56.563 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691898 2024-03-09T13:39:56.564 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691898 2024-03-09T13:39:56.564 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509527 3-493921239043 4-107374182481 5-489626271747 6-150323855426 7-171798691898' 2024-03-09T13:39:56.564 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:56.564 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:39:58.461 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtestt/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528374 2024-03-09T13:39:58.461 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528374 2024-03-09T13:39:58.461 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509526 3-493921239042 4-107374182480 5-489626271746 6-150323855425 7-171798691897 8-193273528374' 2024-03-09T13:39:58.461 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:58.461 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:39:58.461 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500679 2024-03-09T13:39:58.461 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500679 2024-03-09T13:39:58.461 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836584 1-42949673057 2-64424509526 3-493921239042 4-107374182480 5-489626271746 6-150323855425 7-171798691897 8-193273528374 9-459561500679' 2024-03-09T13:39:58.461 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:58.461 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836584 2024-03-09T13:39:58.461 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:58.462 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:39:58.462 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836584 2024-03-09T13:39:58.462 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:58.462 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836584 2024-03-09T13:39:58.462 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2024-03-09T13:39:58.462 INFO:tasks.workunit.client.0.smithi162.stderr:276190: waiting osd.0 seq 21474836584 2024-03-09T13:39:58.462 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:39:58.462 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836583 -lt 21474836584 2024-03-09T13:39:58.462 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:39:58.462 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:39:58.463 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:39:58.463 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836585 -lt 21474836584 2024-03-09T13:39:58.463 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:58.463 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673057 2024-03-09T13:39:58.463 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:58.463 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:39:58.463 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673057 2024-03-09T13:39:58.463 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:58.463 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673057 2024-03-09T13:39:58.463 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673057' 2024-03-09T13:39:58.463 INFO:tasks.workunit.client.0.smithi162.stderr:276190: waiting osd.1 seq 42949673057 2024-03-09T13:39:58.463 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:39:58.463 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673058 -lt 42949673057 2024-03-09T13:39:58.555 INFO:tasks.workunit.client.0.smithi162.stderr:27/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528375 2024-03-09T13:39:58.555 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528375 2024-03-09T13:39:58.556 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509527 3-493921239043 4-107374182481 5-489626271747 6-150323855426 7-171798691898 8-193273528375' 2024-03-09T13:39:58.556 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:39:58.556 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:39:58.556 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500680 2024-03-09T13:39:58.556 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500680 2024-03-09T13:39:58.556 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836585 1-42949673058 2-64424509527 3-493921239043 4-107374182481 5-489626271747 6-150323855426 7-171798691898 8-193273528375 9-459561500680' 2024-03-09T13:39:58.556 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:58.556 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836585 2024-03-09T13:39:58.556 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:58.556 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:39:58.556 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836585 2024-03-09T13:39:58.556 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:58.556 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836585 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: waiting osd.0 seq 21474836585 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836583 -lt 21474836585 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836585 -lt 21474836585 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673058 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673058 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:58.557 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673058 2024-03-09T13:39:58.558 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-03-09T13:39:58.558 INFO:tasks.workunit.client.0.smithi162.stderr:276188: waiting osd.1 seq 42949673058 2024-03-09T13:39:58.558 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:39:58.558 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673058 -lt 42949673058 2024-03-09T13:39:59.370 INFO:tasks.workunit.client.0.smithi162.stderr:2766190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:59.370 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509526 2024-03-09T13:39:59.370 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:59.370 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:39:59.370 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509526 2024-03-09T13:39:59.370 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:59.370 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509526 2024-03-09T13:39:59.371 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-03-09T13:39:59.371 INFO:tasks.workunit.client.0.smithi162.stderr:276190: waiting osd.2 seq 64424509526 2024-03-09T13:39:59.371 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:39:59.371 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509528 -lt 64424509526 2024-03-09T13:39:59.371 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-493921239042 2024-03-09T13:39:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:39:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-493921239042 2024-03-09T13:39:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239042 2024-03-09T13:39:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 493921239042' 2024-03-09T13:39:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:276190: waiting osd.3 seq 493921239042 2024-03-09T13:39:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:39:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239043 -lt 493921239042 2024-03-09T13:39:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182480 2024-03-09T13:39:59.372 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:59.373 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:39:59.373 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182480 2024-03-09T13:39:59.373 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:59.373 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182480 2024-03-09T13:39:59.373 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182480' 2024-03-09T13:39:59.373 INFO:tasks.workunit.client.0.smithi162.stderr:276190: waiting osd.4 seq 107374182480 2024-03-09T13:39:59.373 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:39:59.373 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182482 -lt 107374182480 2024-03-09T13:39:59.373 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:59.373 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-489626271746 2024-03-09T13:39:59.373 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:59.445 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:59.446 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509527 2024-03-09T13:39:59.446 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:59.446 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:39:59.446 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509527 2024-03-09T13:39:59.446 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:59.446 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509527 2024-03-09T13:39:59.446 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509527' 2024-03-09T13:39:59.446 INFO:tasks.workunit.client.0.smithi162.stderr:276188: waiting osd.2 seq 64424509527 2024-03-09T13:39:59.446 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:39:59.446 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509528 -lt 64424509527 2024-03-09T13:39:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-493921239043 2024-03-09T13:39:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:39:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-493921239043 2024-03-09T13:39:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=493921239043 2024-03-09T13:39:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 493921239043' 2024-03-09T13:39:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:276188: waiting osd.3 seq 493921239043 2024-03-09T13:39:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:39:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 493921239043 -lt 493921239043 2024-03-09T13:39:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:59.447 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182481 2024-03-09T13:39:59.448 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:39:59.448 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:39:59.448 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182481 2024-03-09T13:39:59.448 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:39:59.448 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182481 2024-03-09T13:39:59.448 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182481' 2024-03-09T13:39:59.448 INFO:tasks.workunit.client.0.smithi162.stderr:276188: waiting osd.4 seq 107374182481 2024-03-09T13:39:59.448 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:39:59.448 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182482 -lt 107374182481 2024-03-09T13:39:59.448 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:39:59.448 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-489626271747 2024-03-09T13:39:59.448 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:00.310 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22211: flush_pg_stats: osd=5 2024-03-09T13:40:00.310 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-489626271746 2024-03-09T13:40:00.310 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:00.310 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271746 2024-03-09T13:40:00.310 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 489626271746' 2024-03-09T13:40:00.310 INFO:tasks.workunit.client.0.smithi162.stderr:276190: waiting osd.5 seq 489626271746 2024-03-09T13:40:00.310 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:40:00.310 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271748 -lt 489626271746 2024-03-09T13:40:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855425 2024-03-09T13:40:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:40:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855425 2024-03-09T13:40:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855425 2024-03-09T13:40:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855425' 2024-03-09T13:40:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:276190: waiting osd.6 seq 150323855425 2024-03-09T13:40:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:40:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855427 -lt 150323855425 2024-03-09T13:40:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691897 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691897 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691897 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691897' 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: waiting osd.7 seq 171798691897 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691898 -lt 171798691897 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528374 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:40:00.312 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528374 2024-03-09T13:40:00.313 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:00.313 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528374 2024-03-09T13:40:00.324 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.c11: flush_pg_stats: osd=5 2024-03-09T13:40:00.324 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-489626271747 2024-03-09T13:40:00.324 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:00.324 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=489626271747 2024-03-09T13:40:00.324 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 489626271747' 2024-03-09T13:40:00.324 INFO:tasks.workunit.client.0.smithi162.stderr:276188: waiting osd.5 seq 489626271747 2024-03-09T13:40:00.324 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:40:00.325 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 489626271748 -lt 489626271747 2024-03-09T13:40:00.325 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:00.325 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855426 2024-03-09T13:40:00.325 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:00.325 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:40:00.325 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855426 2024-03-09T13:40:00.325 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:00.325 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855426 2024-03-09T13:40:00.325 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855426' 2024-03-09T13:40:00.325 INFO:tasks.workunit.client.0.smithi162.stderr:276188: waiting osd.6 seq 150323855426 2024-03-09T13:40:00.325 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:40:00.325 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855427 -lt 150323855426 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691898 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691898 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691898 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691898' 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: waiting osd.7 seq 171798691898 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691898 -lt 171798691898 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528375 2024-03-09T13:40:00.326 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:00.327 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:40:00.327 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528375 2024-03-09T13:40:00.327 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:00.327 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528375 2024-03-09T13:40:01.895 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.cllient.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528374' 2024-03-09T13:40:01.896 INFO:tasks.workunit.client.0.smithi162.stderr:276190: waiting osd.8 seq 193273528374 2024-03-09T13:40:01.896 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:40:01.896 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528375 -lt 193273528374 2024-03-09T13:40:01.896 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:01.896 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500679 2024-03-09T13:40:01.896 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:01.896 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:40:01.896 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500679 2024-03-09T13:40:01.896 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:01.896 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500679 2024-03-09T13:40:01.897 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500679' 2024-03-09T13:40:01.897 INFO:tasks.workunit.client.0.smithi162.stderr:276190: waiting osd.9 seq 459561500679 2024-03-09T13:40:01.897 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:40:01.897 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500681 -lt 459561500679 2024-03-09T13:40:01.897 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:40:01.897 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:40:01.897 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:40:01.897 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:40:01.897 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:40:01.897 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:40:01.897 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:40:01.897 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:40:01.897 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:40:01.898 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:40:01.898 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:40:01.898 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:40:01.898 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:40:01.898 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:40:01.898 INFO:tasks.workunit.client.0.smithi162.stderr:276190: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:40:01.898 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:40:01.898 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:40:01.898 INFO:tasks.workunit.client.0.smithi162.stderr:276190: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:40:01.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:40:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:ient.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528375' 2024-03-09T13:40:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:276188: waiting osd.8 seq 193273528375 2024-03-09T13:40:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:40:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528375 -lt 193273528375 2024-03-09T13:40:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500680 2024-03-09T13:40:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:40:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500680 2024-03-09T13:40:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500680 2024-03-09T13:40:01.981 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500680' 2024-03-09T13:40:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:276188: waiting osd.9 seq 459561500680 2024-03-09T13:40:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:40:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500681 -lt 459561500680 2024-03-09T13:40:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:40:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:40:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:40:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:40:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:40:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:40:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:40:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:40:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:40:01.982 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:276188: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:276188: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 276187 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:40:01.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:40:01.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: corrupt_and_repair_two: return_code=0 2024-03-09T13:40:01.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:198: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:40:01.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: get_pg ecpool SOMETHING 2024-03-09T13:40:01.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:40:01.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:40:01.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool SOMETHING 2024-03-09T13:40:01.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:40:02.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:203: corrupt_and_repair_two: local pg=2.0 2024-03-09T13:40:02.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:204: corrupt_and_repair_two: repair 2.0 2024-03-09T13:40:02.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T13:40:02.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T13:40:02.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:40:02.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:40:02.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:40:02.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:40:02.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:39:25.537430+0000 2024-03-09T13:40:02.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T13:40:02.787 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.3 to repair 2024-03-09T13:40:02.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T13:39:25.537430+0000 2024-03-09T13:40:02.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T13:40:02.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:39:25.537430+0000 2024-03-09T13:40:02.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:40:02.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:40:02.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:40:02.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:40:02.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:40:02.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:40:02.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:40:02.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:40:03.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:39:25.537430+0000 '>' 2024-03-09T13:39:25.537430+0000 2024-03-09T13:40:03.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:40:04.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:40:04.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:40:04.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:40:04.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:40:04.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:40:04.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:40:04.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:40:04.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:39:25.537430+0000 '>' 2024-03-09T13:39:25.537430+0000 2024-03-09T13:40:04.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:40:05.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:40:05.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:40:05.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T13:40:05.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T13:40:05.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:40:05.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:40:05.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T13:40:05.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:40:03.512388+0000 '>' 2024-03-09T13:39:25.537430+0000 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: corrupt_and_repair_two: pids= 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 279589"' 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 279589' 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2127: run_in_background: local pid_variable=pids 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2128: run_in_background: shift 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: eval 'pids+=" 279591"' 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: run_in_background: pids+=' 279591' 2024-03-09T13:40:05.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:211: corrupt_and_repair_two: wait_background pids 2024-03-09T13:40:05.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: wait_background: pids=' 279589 279591' 2024-03-09T13:40:05.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: wait_background: return_code=0 2024-03-09T13:40:05.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:40:05.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/279592: /' 2024-03-09T13:40:05.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 279589 2024-03-09T13:40:05.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:40:05.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: sed 's/^/279594: /' 2024-03-09T13:40:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:40:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:40:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-03-09T13:40:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:40:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:40:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:40:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:40:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-03-09T13:40:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:40:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2024-03-09T13:40:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:279592: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2024-03-09T13:40:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:279592: _ 2024-03-09T13:40:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:279592: hinfo_key 2024-03-09T13:40:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:279592: snapset 2024-03-09T13:40:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2024-03-09T13:40:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:40:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:40:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-03-09T13:40:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:40:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/3 2024-03-09T13:40:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:40:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:40:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:40:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:40:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2024-03-09T13:40:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2024-03-09T13:40:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:40:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:40:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:40:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:40:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:40:08.638 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:40:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:40:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/3 2024-03-09T13:40:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-03-09T13:40:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:279592: start osd.3 2024-03-09T13:40:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:40:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:40:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-03-09T13:40:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:40:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:40:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:40:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:40:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-03-09T13:40:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:40:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2024-03-09T13:40:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:40:08.943 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:40:08.943 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:40:08.943 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:40:08.943 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:40:08.943 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:40:08.943 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2024-03-09T13:40:08.943 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:40:08.943 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:40:08.943 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-03-09T13:40:08.943 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:40:08.943 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:40:08.943 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2024-03-09T13:40:08.944 INFO:tasks.workunit.client.0.smithi162.stderr:279594: _ 2024-03-09T13:40:08.944 INFO:tasks.workunit.client.0.smithi162.stderr:279594: hinfo_key 2024-03-09T13:40:08.944 INFO:tasks.workunit.client.0.smithi162.stderr:279594: snapset 2024-03-09T13:40:08.944 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2024-03-09T13:40:08.944 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:40:08.944 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:40:08.944 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-03-09T13:40:08.944 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:40:08.944 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/5 2024-03-09T13:40:08.944 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:40:08.945 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:40:08.945 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:40:08.945 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:40:08.945 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2024-03-09T13:40:08.945 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2024-03-09T13:40:08.945 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:40:08.945 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:40:08.945 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:40:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:40:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:40:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:40:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:279594: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:40:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:279594: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:40:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:279594: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:40:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:40:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:40:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:40:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:40:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:40:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:40:08.946 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:40:08.947 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:40:08.947 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:40:08.947 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:40:08.947 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:40:08.947 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/5 2024-03-09T13:40:08.947 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-03-09T13:40:08.947 INFO:tasks.workunit.client.0.smithi162.stderr:279594: start osd.5 2024-03-09T13:40:12.834 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reclen=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:40:12.834 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/3/whoami 2024-03-09T13:40:12.834 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-03-09T13:40:12.834 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:40:12.834 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:40:12.835 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:40:12.835 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 2024-03-09T13:40:08.665+0000 7eff5b145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:12.835 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 2024-03-09T13:40:08.668+0000 7eff5b145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:12.835 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 2024-03-09T13:40:08.669+0000 7eff5b145640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:12.835 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-03-09T13:40:12.835 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:40:12.835 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-03-09T13:40:12.835 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:40:12.835 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:40:12.835 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:12.835 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:40:12.835 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 0 2024-03-09T13:40:12.835 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 2024-03-09T13:40:09.182+0000 7eff5b145640 -1 Falling back to public interface 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 1 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:40:12.836 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 2 2024-03-09T13:40:12.837 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:12.837 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:40:12.837 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 2024-03-09T13:40:11.688+0000 7eff5b145640 -1 osd.3 116 log_to_monitors true 2024-03-09T13:40:12.837 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:12.837 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:12.837 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:12.837 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:40:12.837 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 3 2024-03-09T13:40:12.837 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:13.642 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpeovery_ops 2024-03-09T13:40:13.643 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/5/whoami 2024-03-09T13:40:13.643 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-03-09T13:40:13.643 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:40:13.643 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:40:13.643 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:40:13.643 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 2024-03-09T13:40:08.970+0000 7fe047a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:13.643 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 2024-03-09T13:40:08.991+0000 7fe047a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:13.643 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 2024-03-09T13:40:08.992+0000 7fe047a1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:13.643 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-03-09T13:40:13.643 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:40:13.643 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-03-09T13:40:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:40:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:40:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:40:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 0 2024-03-09T13:40:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:40:13.645 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 2024-03-09T13:40:09.514+0000 7fe047a1b640 -1 Falling back to public interface 2024-03-09T13:40:13.645 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:13.645 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:13.645 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:13.645 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:40:13.645 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 1 2024-03-09T13:40:13.646 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:13.646 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:40:13.646 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:13.646 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:13.646 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:13.646 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:40:13.647 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 2 2024-03-09T13:40:13.647 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:13.647 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:40:13.647 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 2024-03-09T13:40:12.236+0000 7fe047a1b640 -1 osd.5 116 log_to_monitors true 2024-03-09T13:40:13.647 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:13.647 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:13.648 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:13.648 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:40:13.648 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 3 2024-03-09T13:40:13.648 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:13.648 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:40:14.409 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/rs.sh:984: wait_for_osd: grep 'osd.3 up' 2024-03-09T13:40:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:279592: osd.3 up in weight 1 up_from 121 up_thru 121 down_at 117 last_clean_interval [115,116) [v2:127.0.0.1:6826/2660803316,v1:127.0.0.1:6827/2660803316] [v2:127.0.0.1:6828/2660803316,v1:127.0.0.1:6829/2660803316] exists,up 2095e0ae-504c-4658-8991-61158f227389 2024-03-09T13:40:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:40:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:40:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:40:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:40:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:40:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:40:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:40:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:40:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:279592: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:40:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:279592: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:40:14.410 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:40:14.411 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:40:14.411 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:40:14.411 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:40:14.411 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:40:14.411 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:40:14.411 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:40:14.411 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:40:14.411 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:40:14.411 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 1 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 2 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 3 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 4 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 5 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 6 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 7 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 8 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: 9' 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836589 2024-03-09T13:40:14.412 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836589 2024-03-09T13:40:14.413 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589' 2024-03-09T13:40:14.413 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:14.413 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:40:14.413 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673063 2024-03-09T13:40:14.413 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673063 2024-03-09T13:40:14.413 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063' 2024-03-09T13:40:15.552 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:15.552 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:40:15.552 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509532 2024-03-09T13:40:15.552 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509532 2024-03-09T13:40:15.552 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532' 2024-03-09T13:40:15.552 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:15.552 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:40:15.552 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042818 2024-03-09T13:40:15.552 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042818 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-519691042818' 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182486 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182486 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-519691042818 4-107374182486' 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=532575944706 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 532575944706 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-519691042818 4-107374182486 5-532575944706' 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855431 2024-03-09T13:40:15.553 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855431 2024-03-09T13:40:15.554 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-519691042818 4-107374182486 5-532575944706 6-150323855431' 2024-03-09T13:40:15.554 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:15.554 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:40:15.554 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691902 2024-03-09T13:40:15.554 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691902 2024-03-09T13:40:15.554 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-519691042818 4-107374182486 5-532575944706 6-150323855431 7-171798691902' 2024-03-09T13:40:15.554 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:15.972 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:15.972 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:15.972 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:15.972 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:40:15.972 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 4 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: osd.5 up in weight 1 up_from 124 up_thru 124 down_at 117 last_clean_interval [114,116) [v2:127.0.0.1:6874/2554786312,v1:127.0.0.1:6875/2554786312] [v2:127.0.0.1:6876/2554786312,v1:127.0.0.1:6877/2554786312] exists,up 1a7a2a5c-3d7e-4a54-91c1-c81c5be3f535 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:40:15.973 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:40:15.974 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:40:15.974 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:40:15.974 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:40:15.974 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:40:15.974 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:40:15.974 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:40:15.974 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:40:15.974 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:40:15.974 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 1 2024-03-09T13:40:15.974 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 2 2024-03-09T13:40:15.974 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 3 2024-03-09T13:40:15.974 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 4 2024-03-09T13:40:15.975 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 5 2024-03-09T13:40:15.975 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 6 2024-03-09T13:40:15.975 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 7 2024-03-09T13:40:15.975 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 8 2024-03-09T13:40:15.975 INFO:tasks.workunit.client.0.smithi162.stderr:279594: 9' 2024-03-09T13:40:15.975 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:40:15.975 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:15.975 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:40:15.975 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836591 2024-03-09T13:40:15.975 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836591 2024-03-09T13:40:15.975 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591' 2024-03-09T13:40:16.670 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd a/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:40:16.670 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528379 2024-03-09T13:40:16.670 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528379 2024-03-09T13:40:16.670 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-519691042818 4-107374182486 5-532575944706 6-150323855431 7-171798691902 8-193273528379' 2024-03-09T13:40:16.670 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:16.670 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:40:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500685 2024-03-09T13:40:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500685 2024-03-09T13:40:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509532 3-519691042818 4-107374182486 5-532575944706 6-150323855431 7-171798691902 8-193273528379 9-459561500685' 2024-03-09T13:40:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836589 2024-03-09T13:40:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:40:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836589 2024-03-09T13:40:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836589 2024-03-09T13:40:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-03-09T13:40:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:279592: waiting osd.0 seq 21474836589 2024-03-09T13:40:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:40:16.672 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836590 -lt 21474836589 2024-03-09T13:40:16.672 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:16.672 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673063 2024-03-09T13:40:16.672 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:16.672 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:40:16.672 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673063 2024-03-09T13:40:16.672 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:16.672 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673063 2024-03-09T13:40:16.672 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2024-03-09T13:40:16.672 INFO:tasks.workunit.client.0.smithi162.stderr:279592: waiting osd.1 seq 42949673063 2024-03-09T13:40:16.672 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:40:16.672 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673063 -lt 42949673063 2024-03-09T13:40:16.673 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:16.673 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509532 2024-03-09T13:40:16.673 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:16.759 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cepin $ids 2024-03-09T13:40:16.759 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:40:16.759 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673064 2024-03-09T13:40:16.759 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673064 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064' 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509534 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509534 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509534' 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=519691042819 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 519691042819 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509534 3-519691042819' 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-03-09T13:40:16.760 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182487 2024-03-09T13:40:16.761 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182487 2024-03-09T13:40:16.761 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509534 3-519691042819 4-107374182487' 2024-03-09T13:40:16.761 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:16.761 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-03-09T13:40:16.761 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=532575944707 2024-03-09T13:40:16.761 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 532575944707 2024-03-09T13:40:16.761 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509534 3-519691042819 4-107374182487 5-532575944707' 2024-03-09T13:40:16.761 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:16.761 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-03-09T13:40:16.761 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855432 2024-03-09T13:40:16.761 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855432 2024-03-09T13:40:16.761 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509534 3-519691042819 4-107374182487 5-532575944707 6-150323855432' 2024-03-09T13:40:16.761 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:16.762 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-03-09T13:40:17.586 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helperhtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:40:17.586 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509532 2024-03-09T13:40:17.586 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:17.586 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509532 2024-03-09T13:40:17.586 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-03-09T13:40:17.586 INFO:tasks.workunit.client.0.smithi162.stderr:279592: waiting osd.2 seq 64424509532 2024-03-09T13:40:17.586 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509532 -lt 64424509532 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-519691042818 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-519691042818 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=519691042818 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 519691042818' 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: waiting osd.3 seq 519691042818 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042818 -lt 519691042818 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182486 2024-03-09T13:40:17.587 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182486 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182486 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182486' 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: waiting osd.4 seq 107374182486 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182487 -lt 107374182486 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-532575944706 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-532575944706 2024-03-09T13:40:17.588 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:18.627 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=5325s.sh:2202: flush_pg_stats: seq=171798691904 2024-03-09T13:40:18.627 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691904 2024-03-09T13:40:18.627 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509534 3-519691042819 4-107374182487 5-532575944707 6-150323855432 7-171798691904' 2024-03-09T13:40:18.627 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:18.627 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-03-09T13:40:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528381 2024-03-09T13:40:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528381 2024-03-09T13:40:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509534 3-519691042819 4-107374182487 5-532575944707 6-150323855432 7-171798691904 8-193273528381' 2024-03-09T13:40:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:40:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-03-09T13:40:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500686 2024-03-09T13:40:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500686 2024-03-09T13:40:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836591 1-42949673064 2-64424509534 3-519691042819 4-107374182487 5-532575944707 6-150323855432 7-171798691904 8-193273528381 9-459561500686' 2024-03-09T13:40:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836591 2024-03-09T13:40:18.628 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:40:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836591 2024-03-09T13:40:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836591 2024-03-09T13:40:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2024-03-09T13:40:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:279594: waiting osd.0 seq 21474836591 2024-03-09T13:40:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:40:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836590 -lt 21474836591 2024-03-09T13:40:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:40:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:40:18.629 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:40:18.630 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836591 -lt 21474836591 2024-03-09T13:40:18.630 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:18.630 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673064 2024-03-09T13:40:18.630 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:18.630 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:40:18.727 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-429496730675944706 2024-03-09T13:40:18.727 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 532575944706' 2024-03-09T13:40:18.727 INFO:tasks.workunit.client.0.smithi162.stderr:279592: waiting osd.5 seq 532575944706 2024-03-09T13:40:18.727 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:40:18.727 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 532575944707 -lt 532575944706 2024-03-09T13:40:18.727 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:18.727 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855431 2024-03-09T13:40:18.727 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:40:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855431 2024-03-09T13:40:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855431 2024-03-09T13:40:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855431' 2024-03-09T13:40:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:279592: waiting osd.6 seq 150323855431 2024-03-09T13:40:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:40:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855433 -lt 150323855431 2024-03-09T13:40:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691902 2024-03-09T13:40:18.728 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:18.729 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:40:18.729 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691902 2024-03-09T13:40:18.729 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:18.729 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691902 2024-03-09T13:40:18.729 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691902' 2024-03-09T13:40:18.729 INFO:tasks.workunit.client.0.smithi162.stderr:279592: waiting osd.7 seq 171798691902 2024-03-09T13:40:18.729 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:40:18.729 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691904 -lt 171798691902 2024-03-09T13:40:18.729 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:18.729 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528379 2024-03-09T13:40:18.729 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:18.729 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:40:18.729 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528379 2024-03-09T13:40:18.730 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:18.730 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528379 2024-03-09T13:40:18.730 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528379' 2024-03-09T13:40:18.730 INFO:tasks.workunit.client.0.smithi162.stderr:279592: waiting osd.8 seq 193273528379 2024-03-09T13:40:18.730 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:40:19.534 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 14 2024-03-09T13:40:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673064 2024-03-09T13:40:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673064' 2024-03-09T13:40:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:279594: waiting osd.1 seq 42949673064 2024-03-09T13:40:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:40:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673064 -lt 42949673064 2024-03-09T13:40:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509534 2024-03-09T13:40:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:40:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509534 2024-03-09T13:40:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:19.535 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509534 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: waiting osd.2 seq 64424509534 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509534 -lt 64424509534 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-519691042819 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-519691042819 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=519691042819 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 519691042819' 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: waiting osd.3 seq 519691042819 2024-03-09T13:40:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-03-09T13:40:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 519691042820 -lt 519691042819 2024-03-09T13:40:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182487 2024-03-09T13:40:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-03-09T13:40:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182487 2024-03-09T13:40:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182487 2024-03-09T13:40:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182487' 2024-03-09T13:40:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:279594: waiting osd.4 seq 107374182487 2024-03-09T13:40:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubun93273528381 -lt 193273528379 2024-03-09T13:40:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500685 2024-03-09T13:40:20.015 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500685 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500685 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500685' 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: waiting osd.9 seq 459561500685 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500686 -lt 459561500685 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:40:20.016 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:40:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:40:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:40:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:40:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:40:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:40:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:40:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:279592: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:40:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:40:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:40:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:279592: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:40:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:40:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: wait_background: for pid in $pids 2024-03-09T13:40:20.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: wait_background: wait 279591 2024-03-09T13:40:20.695 INFO:tasks.workunit.client.0.smithi162.stderr:tu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-03-09T13:40:20.695 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182488 -lt 107374182487 2024-03-09T13:40:20.695 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:20.695 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-532575944707 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-532575944707 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=532575944707 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 532575944707' 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: waiting osd.5 seq 532575944707 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 532575944708 -lt 532575944707 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855432 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855432 2024-03-09T13:40:20.696 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855432 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855432' 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: waiting osd.6 seq 150323855432 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855433 -lt 150323855432 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691904 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691904 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691904 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691904' 2024-03-09T13:40:20.697 INFO:tasks.workunit.client.0.smithi162.stderr:279594: waiting osd.7 seq 171798691904 2024-03-09T13:40:20.698 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-03-09T13:40:20.698 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691904 -lt 171798691904 2024-03-09T13:40:20.698 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:21.879 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528381 2024-03-09T13:40:21.879 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:21.879 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-03-09T13:40:21.880 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528381 2024-03-09T13:40:21.880 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:21.880 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528381 2024-03-09T13:40:21.880 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528381' 2024-03-09T13:40:21.880 INFO:tasks.workunit.client.0.smithi162.stderr:279594: waiting osd.8 seq 193273528381 2024-03-09T13:40:21.880 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-03-09T13:40:21.880 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528381 -lt 193273528381 2024-03-09T13:40:21.880 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:40:21.880 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-459561500686 2024-03-09T13:40:21.880 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:40:21.880 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-03-09T13:40:21.880 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-459561500686 2024-03-09T13:40:21.880 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:40:21.881 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500686 2024-03-09T13:40:21.881 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 459561500686' 2024-03-09T13:40:21.881 INFO:tasks.workunit.client.0.smithi162.stderr:279594: waiting osd.9 seq 459561500686 2024-03-09T13:40:21.881 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-03-09T13:40:21.881 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500687 -lt 459561500686 2024-03-09T13:40:21.881 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:40:21.881 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:40:21.881 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:40:21.881 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T13:40:21.881 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:40:21.881 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:40:21.881 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:40:21.881 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:40:21.882 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:40:21.882 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:40:21.882 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:40:21.882 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:40:21.882 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:40:22.220 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:40:22.220 INFO:tasks.workunit.client.0.smithi162.stderr:279594: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:40:22.220 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T13:40:22.220 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:40:22.220 INFO:tasks.workunit.client.0.smithi162.stderr:279594: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:40:22.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: run_in_background: return 0 2024-03-09T13:40:22.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: eval 'pids='\'''\''' 2024-03-09T13:40:22.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: wait_background: pids= 2024-03-09T13:40:22.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: wait_background: return 0 2024-03-09T13:40:22.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:212: corrupt_and_repair_two: return_code=0 2024-03-09T13:40:22.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:213: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2024-03-09T13:40:22.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:40:22.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:40:22.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:40:22.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:40:22.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:40:22.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:40:22.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:40:22.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:40:22.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:40:22.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:40:22.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:40:22.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:40:22.384 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:40:22.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:40:22.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:40:22.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:40:22.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:40:22.386 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:40:22.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:40:22.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:40:22.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:40:22.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:40:22.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:40:22.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:40:22.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:40:22.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:40:22.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:40:22.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:40:22.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:40:22.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:40:22.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:40:22.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:40:22.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:40:22.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:40:22.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:40:22.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:40:22.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:40:22.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:40:22.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:40:22.453 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:40:22.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:40:22.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:40:22.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:40:22.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:40:22.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:40:22.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:40:22.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:40:22.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:40:22.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:40:22.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:40:22.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:40:22.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:40:22.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:40:22.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:40:22.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:40:22.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:40:22.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:40:22.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:40:22.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:40:22.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:40:22.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:40:22.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:40:22.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:40:22.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:40:22.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:40:22.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:40:22.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:40:22.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:40:22.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:40:22.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:40:22.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:40:22.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:40:22.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2024-03-09T13:40:22.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2024-03-09T13:40:22.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2024-03-09T13:40:22.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-09T13:40:22.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:40:22.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:40:22.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:40:22.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:40:22.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:40:22.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-09T13:40:22.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:40:22.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:40:22.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:40:22.936 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:40:22.936 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:40:22.936 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:40:22.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:40:22.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-09T13:40:23.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:40:23.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:40:23.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:40:23.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:40:23.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:40:23.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:40:23.106 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:40:23.106 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:40:23.106 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:40:23.106 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:40:23.106 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:40:23.106 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:40:23.106 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:40:23.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:40:23.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:40:23.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:40:23.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:40:23.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:40:23.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:40:23.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:40:23.187 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:40:23.187 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:40:23.187 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:40:23.187 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:40:23.187 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:40:23.187 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:40:23.188 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:40:23.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:40:23.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:40:23.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2024-03-09T13:40:23.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:40:23.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:40:23.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:40:23.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:40:23.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:40:23.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:40:24.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:40:24.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:40:24.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:40:24.081 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:40:24.081 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:40:24.081 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:40:24.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:40:24.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:40:24.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:40:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2024-03-09T13:40:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:40:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:40:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:40:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:40:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:40:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:40:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:40:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:40:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:40:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:40:24.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:40:24.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:40:24.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:40:24.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:40:24.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:40:24.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:40:24.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:40:24.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:40:24.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:40:24.118 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:40:24.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:40:24.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:40:24.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:40:24.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:40:24.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:40:24.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:40:24.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:40:24.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:40:24.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:40:24.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:40:24.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:40:24.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:40:24.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:40:24.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:40:24.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:40:24.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9045fb85-17e5-4e4c-bafe-dc58a7365987 2024-03-09T13:40:24.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9045fb85-17e5-4e4c-bafe-dc58a7365987' 2024-03-09T13:40:24.120 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 9045fb85-17e5-4e4c-bafe-dc58a7365987 2024-03-09T13:40:24.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:40:24.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDIZuxlRbTLBxAAGS/vHrUC7NLRq+A2JixK5Q== 2024-03-09T13:40:24.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDIZuxlRbTLBxAAGS/vHrUC7NLRq+A2JixK5Q=="}' 2024-03-09T13:40:24.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9045fb85-17e5-4e4c-bafe-dc58a7365987 -i td/osd-scrub-repair/0/new.json 2024-03-09T13:40:24.336 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:40:24.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:40:24.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIZuxlRbTLBxAAGS/vHrUC7NLRq+A2JixK5Q== --osd-uuid 9045fb85-17e5-4e4c-bafe-dc58a7365987 2024-03-09T13:40:24.374 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:24.373+0000 7f141d415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:24.386 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:24.385+0000 7f141d415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:24.387 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:24.386+0000 7f141d415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:24.388 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:24.387+0000 7f141d415640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:40:24.388 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:24.387+0000 7f141d415640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:40:24.388 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:24.387+0000 7f141d415640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:40:24.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:24.477+0000 7f141d415640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:40:30.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:40:30.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:40:30.023 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:40:30.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:40:30.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:40:30.378 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:40:30.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:40:30.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:40:30.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:40:30.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:40:30.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:40:30.404 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:30.403+0000 7f953c2bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:30.406 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:30.405+0000 7f953c2bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:30.407 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:30.406+0000 7f953c2bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:30.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:40:30.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:40:30.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:40:30.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:40:30.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:40:30.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:30.651 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:40:30.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:40:30.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:30.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:30.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:30.927 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:30.926+0000 7f953c2bd640 -1 Falling back to public interface 2024-03-09T13:40:31.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:31.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:31.949 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:40:31.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:40:31.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:31.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:32.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:33.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:33.210 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:40:33.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:40:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:33.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:33.262 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:33.260+0000 7f953c2bd640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:40:33.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:34.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:34.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:34.513 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:40:34.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:40:34.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:34.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:34.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:35.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:35.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:35.810 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:40:35.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:40:35.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:35.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:36.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:37.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:37.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:37.156 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:40:37.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:40:37.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:37.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:37.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:38.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:38.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:38.442 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:40:38.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:40:38.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:38.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:38.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:39.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:39.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:39.818 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:40:39.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:40:39.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:39.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:40.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:41.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:41.195 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:40:41.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:40:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:41.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:41.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:42.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:42.483 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:40:42.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:40:42.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:42.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:42.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:43.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:43.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:43.819 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:40:43.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:40:43.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:43.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:44.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:45.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:45.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:45.207 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:40:45.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:40:45.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:45.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:45.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:46.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:46.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:46.571 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:40:46.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:40:46.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:46.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:47.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:48.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:48.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:48.111 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:40:48.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:40:48.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:48.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:48.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:49.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:49.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:49.949 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:40:49.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:40:49.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:49.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:51.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:52.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:52.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:52.045 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:40:52.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:40:52.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:52.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:52.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:53.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:53.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:53.661 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:40:53.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:40:53.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:53.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:54.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:40:55.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:40:55.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:40:55.184 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:40:55.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:40:55.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:40:55.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:40:55.496 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1602833177,v1:127.0.0.1:6803/1602833177] [v2:127.0.0.1:6804/1602833177,v1:127.0.0.1:6805/1602833177] exists,up 9045fb85-17e5-4e4c-bafe-dc58a7365987 2024-03-09T13:40:55.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:40:55.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:40:55.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:40:55.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2024-03-09T13:40:55.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:40:55.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:40:55.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:40:55.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:40:55.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:40:55.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:40:55.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:40:55.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:40:55.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:40:55.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:40:55.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:40:55.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:40:55.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:40:55.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:40:55.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:40:55.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:40:55.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:40:55.502 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:40:55.502 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:40:55.502 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:40:55.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:40:55.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:40:55.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:40:55.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:40:55.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:40:55.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:40:55.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:40:55.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:40:55.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:40:55.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:40:55.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:40:55.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:40:55.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:40:55.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:40:55.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:40:55.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e09a85d9-1d47-4dd6-bedf-10c1f0ce93d2 2024-03-09T13:40:55.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e09a85d9-1d47-4dd6-bedf-10c1f0ce93d2' 2024-03-09T13:40:55.505 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 e09a85d9-1d47-4dd6-bedf-10c1f0ce93d2 2024-03-09T13:40:55.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:40:55.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDnZuxlI5vMHhAAfAZ2aeIeL9flz0dRwXTbCw== 2024-03-09T13:40:55.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDnZuxlI5vMHhAAfAZ2aeIeL9flz0dRwXTbCw=="}' 2024-03-09T13:40:55.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e09a85d9-1d47-4dd6-bedf-10c1f0ce93d2 -i td/osd-scrub-repair/1/new.json 2024-03-09T13:40:55.884 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:40:55.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:40:55.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnZuxlI5vMHhAAfAZ2aeIeL9flz0dRwXTbCw== --osd-uuid e09a85d9-1d47-4dd6-bedf-10c1f0ce93d2 2024-03-09T13:40:55.924 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:55.924+0000 7f3c7a099640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:55.969 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:55.969+0000 7f3c7a099640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:55.970 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:55.970+0000 7f3c7a099640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:40:55.970 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:55.970+0000 7f3c7a099640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:40:55.970 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:55.970+0000 7f3c7a099640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:40:55.970 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:55.970+0000 7f3c7a099640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:40:56.044 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:40:56.043+0000 7f3c7a099640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:41:01.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:41:01.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:41:01.571 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:41:01.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:41:01.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:41:01.928 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:41:01.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:41:01.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:41:01.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:41:01.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:41:01.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:41:01.954 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:01.954+0000 7f788f1df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:41:01.956 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:01.956+0000 7f788f1df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:41:01.957 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:01.957+0000 7f788f1df640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:41:02.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:41:02.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:41:02.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:41:02.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:41:02.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:41:02.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:02.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:41:02.208 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:41:02.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:02.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:02.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:02.476+0000 7f788f1df640 -1 Falling back to public interface 2024-03-09T13:41:02.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:03.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:03.481 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:41:03.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:03.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:41:03.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:03.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:03.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:04.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:04.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:04.767 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:41:04.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:41:04.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:04.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:04.834 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:04.834+0000 7f788f1df640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:41:05.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:06.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:06.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:06.045 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:41:06.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:41:06.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:06.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:06.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:07.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:07.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:07.330 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:41:07.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:41:07.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:07.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:07.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:08.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:08.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:08.640 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:41:08.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:41:08.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:08.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:08.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:09.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:09.929 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:41:09.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:09.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:41:09.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:09.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:10.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:11.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:11.218 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:41:11.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:11.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:41:11.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:11.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:11.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:12.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:12.504 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:41:12.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:12.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:41:12.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:12.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:12.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:13.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:13.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:13.864 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:41:13.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:41:13.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:13.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:14.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:15.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:15.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:15.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:41:15.149 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:41:15.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:15.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:15.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:16.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:16.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:16.435 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:41:16.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:41:16.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:16.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:16.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:17.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:17.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:17.850 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:41:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:41:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:18.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:19.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:19.356 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:41:19.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:19.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:41:19.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:19.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:20.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:21.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:21.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:21.067 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:41:21.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:41:21.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:21.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:21.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:22.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:22.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:22.861 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:41:22.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:41:22.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:22.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:23.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:24.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:24.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:24.564 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:41:24.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:41:24.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:24.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:25.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:26.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:26.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:26.467 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:41:26.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:41:26.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:26.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:27.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:28.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:28.203 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:41:28.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:28.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:41:28.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:28.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:28.519 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1441916750,v1:127.0.0.1:6811/1441916750] [v2:127.0.0.1:6812/1441916750,v1:127.0.0.1:6813/1441916750] exists,up e09a85d9-1d47-4dd6-bedf-10c1f0ce93d2 2024-03-09T13:41:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:41:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:41:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:41:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2024-03-09T13:41:28.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T13:41:28.781 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T13:41:28.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T13:41:28.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T13:41:29.189 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T13:41:29.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:41:30.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T13:41:31.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2024-03-09T13:41:31.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:41:31.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:41:31.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:41:31.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:41:31.894 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:41:31.894 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:41:31.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:41:31.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:41:31.894 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:41:31.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:41:31.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:41:31.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:41:31.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:41:31.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:41:31.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:41:32.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:41:32.264 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:41:32.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:41:32.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:41:32.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:41:32.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-03-09T13:41:32.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-03-09T13:41:32.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-03-09T13:41:32.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:41:32.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:41:32.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-03-09T13:41:32.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-03-09T13:41:32.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672969' 2024-03-09T13:41:32.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:41:32.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-03-09T13:41:32.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:41:32.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:41:32.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-03-09T13:41:32.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:41:32.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-03-09T13:41:32.509 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836492 2024-03-09T13:41:32.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-03-09T13:41:32.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:41:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-03-09T13:41:32.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:41:33.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:41:33.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:41:34.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-03-09T13:41:34.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:41:35.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:41:35.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:41:35.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836492 2024-03-09T13:41:35.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:41:35.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-03-09T13:41:35.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:41:35.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:41:35.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-03-09T13:41:35.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:41:35.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-03-09T13:41:35.353 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672969 2024-03-09T13:41:35.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-03-09T13:41:35.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:41:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672970 -lt 42949672969 2024-03-09T13:41:35.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:41:35.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:41:35.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:41:35.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T13:41:35.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:41:35.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:41:35.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:41:35.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:41:35.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:41:35.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:41:35.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:41:36.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:41:36.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:41:36.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:41:36.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:41:36.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T13:41:36.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:41:36.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:41:36.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2024-03-09T13:41:36.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:41:36.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2024-03-09T13:41:36.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2024-03-09T13:41:36.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:41:36.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:41:36.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:41:36.972 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:41:36.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:41:37.465 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:41:37.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:41:37.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:41:37.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2024-03-09T13:41:37.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2024-03-09T13:41:37.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-03-09T13:41:37.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=SOMETHING 2024-03-09T13:41:37.696 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd SOMETHING 2024-03-09T13:41:37.696 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-09T13:41:37.696 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:41:37.696 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-09T13:41:37.697 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:41:37.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T13:41:37.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd SOMETHING 2024-03-09T13:41:37.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T13:41:38.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2024-03-09T13:41:38.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=0 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:41:38.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:41:38.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:41:38.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:41:38.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:41:38.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:41:38.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:41:38.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2024-03-09T13:41:38.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:41:38.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:41:38.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:41:38.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:41:38.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:41:38.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2024-03-09T13:41:40.037 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-09T13:41:40.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:41:40.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:41:40.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:41:40.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:41:40.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:41:40.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:41:40.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:41:40.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:41:40.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:41:40.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:41:40.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:41:40.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:41:40.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:41:40.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:41:40.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:41:40.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:41:40.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:41:40.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:41:40.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:41:40.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:41:40.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:41:40.558 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:41:40.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:41:40.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:41:40.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:41:40.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:41:40.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:41:40.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:41:40.585 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:40.584+0000 7f1cb6a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:41:40.586 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:40.586+0000 7f1cb6a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:41:40.588 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:40.588+0000 7f1cb6a0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:41:40.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:41:40.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:41:40.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:41:40.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:41:40.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:41:40.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:40.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:41:40.841 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:41:40.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:40.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:41:41.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:41.129 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:41.129+0000 7f1cb6a0d640 -1 Falling back to public interface 2024-03-09T13:41:42.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:42.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:42.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:41:42.114 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:41:42.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:42.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:41:42.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:43.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:43.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:43.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:41:43.398 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:41:43.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:43.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:41:43.488 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:43.487+0000 7f1cb6a0d640 -1 osd.0 21 log_to_monitors true 2024-03-09T13:41:43.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:44.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:44.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:44.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:41:44.698 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:41:44.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:44.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:41:45.014 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 26 up_thru 26 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/2540499413,v1:127.0.0.1:6803/2540499413] [v2:127.0.0.1:6804/2540499413,v1:127.0.0.1:6805/2540499413] exists,up 9045fb85-17e5-4e4c-bafe-dc58a7365987 2024-03-09T13:41:45.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:41:45.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:41:45.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:41:45.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:41:45.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:41:45.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:41:45.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:41:45.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:41:45.015 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:41:45.015 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:41:45.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:41:45.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:41:45.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:41:45.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:41:45.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:41:45.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:41:45.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:41:45.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:41:45.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:41:45.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:41:45.370 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:41:45.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:41:45.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:41:45.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:41:45.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149698 2024-03-09T13:41:45.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149698 2024-03-09T13:41:45.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698' 2024-03-09T13:41:45.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:41:45.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:41:45.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-03-09T13:41:45.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-03-09T13:41:45.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149698 1-42949672973' 2024-03-09T13:41:45.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:41:45.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149698 2024-03-09T13:41:45.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:41:45.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:41:45.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149698 2024-03-09T13:41:45.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:41:45.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149698 2024-03-09T13:41:45.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149698' 2024-03-09T13:41:45.581 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 111669149698 2024-03-09T13:41:45.581 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:41:45.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 111669149698 2024-03-09T13:41:45.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:41:46.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:41:46.851 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:41:47.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149698 -lt 111669149698 2024-03-09T13:41:47.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:41:47.130 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-03-09T13:41:47.130 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:41:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:41:47.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-03-09T13:41:47.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:41:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-03-09T13:41:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-03-09T13:41:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672973 2024-03-09T13:41:47.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:41:47.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-03-09T13:41:47.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:41:47.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:41:47.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:41:47.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T13:41:47.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:41:47.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:41:47.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:41:47.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:41:47.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:41:47.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:41:47.782 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:41:48.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:41:48.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:41:48.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:41:48.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:41:48.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T13:41:48.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:41:48.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:41:48.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-09T13:41:48.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-09T13:41:48.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:41:48.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-09T13:41:48.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:41:48.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-09T13:41:48.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-09T13:41:48.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-09T13:41:48.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-09T13:41:48.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T13:41:48.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:41:48.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:41:48.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T13:41:48.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:41:29.152850+0000 2024-03-09T13:41:48.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-09T13:41:49.199 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-09T13:41:49.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-09T13:41:29.152850+0000 2024-03-09T13:41:49.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-09T13:41:49.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:41:29.152850+0000 2024-03-09T13:41:49.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:41:49.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:41:49.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:41:49.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T13:41:49.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T13:41:49.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:41:49.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:41:49.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T13:41:49.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:41:29.152850+0000 '>' 2024-03-09T13:41:29.152850+0000 2024-03-09T13:41:49.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:41:50.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:41:50.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:41:50.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T13:41:50.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T13:41:50.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:41:50.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:41:50.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T13:41:50.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:41:29.152850+0000 '>' 2024-03-09T13:41:29.152850+0000 2024-03-09T13:41:50.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:41:51.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:41:51.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:41:51.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T13:41:51.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T13:41:51.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:41:51.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:41:51.762 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T13:41:52.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:41:29.152850+0000 '>' 2024-03-09T13:41:29.152850+0000 2024-03-09T13:41:52.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:41:53.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:41:53.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:41:53.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T13:41:53.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T13:41:53.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:41:53.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:41:53.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T13:41:53.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:41:29.152850+0000 '>' 2024-03-09T13:41:29.152850+0000 2024-03-09T13:41:53.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:41:54.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:41:54.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:41:54.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T13:41:54.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T13:41:54.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:41:54.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:41:54.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T13:41:54.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:41:49.703640+0000 '>' 2024-03-09T13:41:29.152850+0000 2024-03-09T13:41:54.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:41:54.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:41:54.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:41:54.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:41:54.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:41:54.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:41:54.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:41:54.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:41:54.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:41:54.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:41:54.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:41:54.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:41:54.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:41:54.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:41:54.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:41:54.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:41:54.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:41:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:41:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:41:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:41:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:41:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:41:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:41:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:41:54.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T13:41:56.032 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-09T13:41:56.033 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:41:56.033 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:41:56.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:41:56.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:41:56.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:41:56.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:41:56.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:41:56.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:41:56.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:41:56.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:41:56.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:41:56.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:41:56.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:41:56.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:41:56.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:41:56.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:41:56.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:41:56.311 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:41:56.311 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:41:56.311 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:41:56.311 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:41:56.311 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:41:56.311 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:41:56.312 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:41:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:41:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:41:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:41:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:41:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:41:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:41:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:41:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:41:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:41:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:41:56.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:41:56.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:41:56.313 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:41:56.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:41:56.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:41:56.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:41:56.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:41:56.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:41:56.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:41:56.339 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:56.338+0000 7f638f011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:41:56.341 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:56.340+0000 7f638f011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:41:56.342 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:56.341+0000 7f638f011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:41:56.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:41:56.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:41:56.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:41:56.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:41:56.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:41:56.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:56.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:41:56.597 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:41:56.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:56.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:56.863 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:56.862+0000 7f638f011640 -1 Falling back to public interface 2024-03-09T13:41:56.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:57.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:57.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:57.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:41:57.870 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:41:57.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:57.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:58.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:41:58.970 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:41:58.968+0000 7f638f011640 -1 osd.1 28 log_to_monitors true 2024-03-09T13:41:59.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:41:59.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:41:59.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:41:59.145 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:41:59.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:41:59.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:41:59.514 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 32 up_thru 26 down_at 29 last_clean_interval [10,28) [v2:127.0.0.1:6810/3068172494,v1:127.0.0.1:6811/3068172494] [v2:127.0.0.1:6812/3068172494,v1:127.0.0.1:6813/3068172494] exists,up e09a85d9-1d47-4dd6-bedf-10c1f0ce93d2 2024-03-09T13:41:59.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:41:59.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:41:59.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:41:59.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:41:59.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:41:59.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:41:59.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:41:59.515 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:41:59.515 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:41:59.515 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:41:59.515 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:41:59.515 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:41:59.516 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:41:59.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:41:59.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:41:59.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:41:59.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:41:59.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:41:59.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:41:59.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:41:59.908 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:41:59.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:41:59.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:41:59.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:42:00.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-03-09T13:42:00.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-03-09T13:42:00.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702' 2024-03-09T13:42:00.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:42:00.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:42:00.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=137438953474 2024-03-09T13:42:00.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 137438953474 2024-03-09T13:42:00.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149702 1-137438953474' 2024-03-09T13:42:00.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:42:00.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149702 2024-03-09T13:42:00.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:42:00.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:42:00.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149702 2024-03-09T13:42:00.192 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:42:00.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-03-09T13:42:00.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149702' 2024-03-09T13:42:00.194 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 111669149702 2024-03-09T13:42:00.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:42:00.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-03-09T13:42:00.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:42:00.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-137438953474 2024-03-09T13:42:00.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:42:00.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:42:00.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-137438953474 2024-03-09T13:42:00.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:42:00.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=137438953474 2024-03-09T13:42:00.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 137438953474' 2024-03-09T13:42:00.561 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 137438953474 2024-03-09T13:42:00.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:42:00.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 137438953474 2024-03-09T13:42:00.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:42:01.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:42:01.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:42:02.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 137438953474 2024-03-09T13:42:02.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:42:03.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:42:03.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:42:03.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 137438953474 -lt 137438953474 2024-03-09T13:42:03.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:42:03.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:42:03.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:42:03.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T13:42:03.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:42:03.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:42:03.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:42:03.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:42:03.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:42:03.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:42:03.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:42:04.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-09T13:42:04.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:42:04.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:42:04.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:42:04.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 4 2024-03-09T13:42:04.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-09T13:42:04.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:42:04.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-09T13:42:04.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:42:04.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:42:04.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:42:04.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:42:04.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:42:04.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:42:04.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:42:04.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:42:04.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:42:04.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:42:04.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:42:04.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:42:04.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:42:04.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:42:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T13:42:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:42:05.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:42:05.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:42:05.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:42:05.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2024-03-09T13:42:05.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-03-09T13:42:05.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=SOMETHING 2024-03-09T13:42:05.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd SOMETHING 2024-03-09T13:42:05.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:42:05.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:226: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:227: corrupt_and_repair_one: local poolname=rbd 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: corrupt_and_repair_one: local osd=1 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:42:05.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:42:05.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:42:05.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:42:05.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:42:05.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:42:05.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:42:05.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:42:05.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2024-03-09T13:42:05.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:42:05.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:42:05.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:42:05.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:42:05.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:42:05.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2024-03-09T13:42:07.009 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:eb822e21:::SOMETHING:head# 2024-03-09T13:42:07.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:42:07.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:42:07.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:42:07.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:42:07.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:42:07.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:42:07.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:42:07.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:42:07.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:42:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:42:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:42:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:42:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:42:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:42:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:42:07.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:42:07.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:42:07.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:42:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:42:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:42:07.529 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:42:07.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:42:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:42:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:42:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:42:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:42:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:42:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:42:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:42:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:42:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:42:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:42:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:42:07.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:42:07.531 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:42:07.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:42:07.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:42:07.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:42:07.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:42:07.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:42:07.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:42:07.557 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:07.556+0000 7f2e9b6c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:42:07.559 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:07.558+0000 7f2e9b6c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:42:07.560 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:07.559+0000 7f2e9b6c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:42:07.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:42:07.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:42:07.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:42:07.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:42:07.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:42:07.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:07.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:42:07.815 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:42:07.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:07.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:42:08.078 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:08.077+0000 7f2e9b6c4640 -1 Falling back to public interface 2024-03-09T13:42:08.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:09.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:09.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:09.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:42:09.085 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:42:09.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:09.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:42:09.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:10.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:10.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:10.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:42:10.360 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:42:10.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:10.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:42:10.490 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:10.489+0000 7f2e9b6c4640 -1 osd.1 34 log_to_monitors true 2024-03-09T13:42:10.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:11.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:11.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:11.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:42:11.760 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:42:11.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:11.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:42:12.039 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [32,34) [v2:127.0.0.1:6810/2791659438,v1:127.0.0.1:6811/2791659438] [v2:127.0.0.1:6812/2791659438,v1:127.0.0.1:6813/2791659438] exists,up e09a85d9-1d47-4dd6-bedf-10c1f0ce93d2 2024-03-09T13:42:12.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:42:12.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:42:12.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:42:12.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:42:12.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:42:12.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:42:12.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:42:12.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:42:12.040 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:42:12.040 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:42:12.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:42:12.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:42:12.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:42:12.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:42:12.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:42:12.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:42:12.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:42:12.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:42:12.126 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:42:12.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:42:12.405 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:42:12.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:42:12.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:42:12.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:42:12.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149705 2024-03-09T13:42:12.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149705 2024-03-09T13:42:12.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149705' 2024-03-09T13:42:12.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:42:12.510 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:42:12.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757250 2024-03-09T13:42:12.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757250 2024-03-09T13:42:12.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149705 1-163208757250' 2024-03-09T13:42:12.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:42:12.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149705 2024-03-09T13:42:12.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:42:12.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:42:12.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149705 2024-03-09T13:42:12.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:42:12.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149705 2024-03-09T13:42:12.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149705' 2024-03-09T13:42:12.618 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 111669149705 2024-03-09T13:42:12.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:42:12.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149704 -lt 111669149705 2024-03-09T13:42:12.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:42:13.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:42:13.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:42:14.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149704 -lt 111669149705 2024-03-09T13:42:14.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:42:15.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:42:15.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:42:15.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149706 -lt 111669149705 2024-03-09T13:42:15.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:42:15.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-163208757250 2024-03-09T13:42:15.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:42:15.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:42:15.462 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-163208757250 2024-03-09T13:42:15.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:42:15.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757250 2024-03-09T13:42:15.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 163208757250' 2024-03-09T13:42:15.464 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 163208757250 2024-03-09T13:42:15.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:42:15.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757250 -lt 163208757250 2024-03-09T13:42:15.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:42:15.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:42:15.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:42:16.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T13:42:16.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:42:16.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:42:16.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:42:16.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:42:16.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:42:16.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:42:16.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:42:16.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:42:16.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:42:16.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:42:16.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:42:16.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T13:42:16.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:42:16.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:42:16.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: get_pg rbd SOMETHING 2024-03-09T13:42:16.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-03-09T13:42:16.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=SOMETHING 2024-03-09T13:42:16.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd SOMETHING 2024-03-09T13:42:16.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:42:17.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:237: corrupt_and_repair_one: local pg=1.3 2024-03-09T13:42:17.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:238: corrupt_and_repair_one: repair 1.3 2024-03-09T13:42:17.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.3 2024-03-09T13:42:17.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.3 2024-03-09T13:42:17.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T13:42:17.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:42:17.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:42:17.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T13:42:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:41:49.703640+0000 2024-03-09T13:42:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.3 2024-03-09T13:42:17.561 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 1.3 on osd.1 to repair 2024-03-09T13:42:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.3 2024-03-09T13:41:49.703640+0000 2024-03-09T13:42:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.3 2024-03-09T13:42:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:41:49.703640+0000 2024-03-09T13:42:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:42:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:42:17.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:42:17.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T13:42:17.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T13:42:17.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:42:17.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:42:17.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T13:42:17.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:41:49.703640+0000 '>' 2024-03-09T13:41:49.703640+0000 2024-03-09T13:42:17.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:42:18.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:42:18.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:42:18.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T13:42:18.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T13:42:18.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:42:18.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:42:18.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T13:42:19.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:41:49.703640+0000 '>' 2024-03-09T13:41:49.703640+0000 2024-03-09T13:42:19.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:42:20.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:42:20.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:42:20.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T13:42:20.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T13:42:20.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:42:20.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:42:20.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T13:42:20.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:41:49.703640+0000 '>' 2024-03-09T13:41:49.703640+0000 2024-03-09T13:42:20.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:42:21.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:42:21.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:42:21.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T13:42:21.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T13:42:21.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:42:21.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:42:21.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T13:42:21.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:41:49.703640+0000 '>' 2024-03-09T13:41:49.703640+0000 2024-03-09T13:42:21.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:42:22.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:42:22.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:42:22.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2024-03-09T13:42:22.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.3 2024-03-09T13:42:22.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:42:22.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:42:22.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2024-03-09T13:42:22.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:42:18.608238+0000 '>' 2024-03-09T13:41:49.703640+0000 2024-03-09T13:42:22.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:42:22.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:242: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:42:22.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:42:22.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:42:23.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:42:23.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2024-03-09T13:42:23.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:42:23.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:42:23.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:42:23.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:42:23.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:42:23.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2024-03-09T13:42:24.459 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2024-03-09T13:42:24.460 INFO:tasks.workunit.client.0.smithi162.stdout:_ 2024-03-09T13:42:24.460 INFO:tasks.workunit.client.0.smithi162.stdout:snapset 2024-03-09T13:42:24.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:42:24.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:42:24.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:42:24.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:42:24.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:42:24.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:42:24.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:42:24.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:42:24.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:42:24.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:42:24.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:42:24.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:42:24.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:42:24.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:42:24.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:42:24.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:42:24.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:42:24.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:42:24.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:42:24.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:42:24.732 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:42:24.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:42:24.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:42:24.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:42:24.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:42:24.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:42:24.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:42:24.759 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:24.759+0000 7f45cfe15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:42:24.761 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:24.761+0000 7f45cfe15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:42:24.762 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:24.762+0000 7f45cfe15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:42:25.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:42:25.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:42:25.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:42:25.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:42:25.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:42:25.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:25.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:42:25.014 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:42:25.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:25.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:42:25.286 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:25.286+0000 7f45cfe15640 -1 Falling back to public interface 2024-03-09T13:42:25.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:26.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:26.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:26.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:42:26.289 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:42:26.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:26.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:42:26.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:27.470 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:27.467+0000 7f45cfe15640 -1 osd.1 39 log_to_monitors true 2024-03-09T13:42:27.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:27.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:27.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:42:27.572 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:42:27.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:27.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:42:27.853 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 43 up_thru 38 down_at 40 last_clean_interval [38,39) [v2:127.0.0.1:6810/2978636396,v1:127.0.0.1:6811/2978636396] [v2:127.0.0.1:6812/2978636396,v1:127.0.0.1:6813/2978636396] exists,up e09a85d9-1d47-4dd6-bedf-10c1f0ce93d2 2024-03-09T13:42:27.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:42:27.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:42:27.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:42:27.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:42:27.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:42:27.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:42:27.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:42:27.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:42:27.854 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:42:27.854 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:42:27.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:42:27.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:42:27.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:42:27.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:42:27.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:42:27.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:42:27.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:42:27.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:42:27.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:42:28.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:42:28.416 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:42:28.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:42:28.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:42:28.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:42:28.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149709 2024-03-09T13:42:28.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149709 2024-03-09T13:42:28.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149709' 2024-03-09T13:42:28.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:42:28.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:42:28.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593730 2024-03-09T13:42:28.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593730 2024-03-09T13:42:28.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-111669149709 1-184683593730' 2024-03-09T13:42:28.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:42:28.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-111669149709 2024-03-09T13:42:28.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:42:28.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:42:28.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-111669149709 2024-03-09T13:42:28.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:42:28.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149709 2024-03-09T13:42:28.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 111669149709' 2024-03-09T13:42:28.714 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 111669149709 2024-03-09T13:42:28.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:42:28.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149708 -lt 111669149709 2024-03-09T13:42:28.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:42:30.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:42:30.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:42:30.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149710 -lt 111669149709 2024-03-09T13:42:30.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:42:30.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593730 2024-03-09T13:42:30.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:42:30.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:42:30.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593730 2024-03-09T13:42:30.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:42:30.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593730 2024-03-09T13:42:30.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2024-03-09T13:42:30.309 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 184683593730 2024-03-09T13:42:30.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:42:30.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593730 -lt 184683593730 2024-03-09T13:42:30.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:42:30.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:42:30.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:42:30.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T13:42:30.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:42:30.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:42:30.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:42:30.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:42:30.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:42:30.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:42:30.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:42:31.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-09T13:42:31.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:42:31.200 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:42:31.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:42:31.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 4 2024-03-09T13:42:31.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-03-09T13:42:31.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:42:31.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-03-09T13:42:31.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:42:31.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:42:31.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:42:31.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:42:31.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:42:31.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:42:31.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:42:31.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:42:31.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:42:31.645 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:42:31.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-09T13:42:31.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:42:31.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:42:31.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:42:32.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 4 2024-03-09T13:42:32.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-03-09T13:42:32.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:42:32.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:42:32.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:42:32.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:42:32.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:42:32.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:42:32.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:42:32.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:42:32.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:42:32.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:42:32.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:42:32.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:42:32.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:42:32.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:42:32.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:42:32.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:42:32.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:42:32.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:42:32.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:42:32.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:42:33.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-03-09T13:42:33.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:42:33.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:42:33.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:42:33.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 4 2024-03-09T13:42:33.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-03-09T13:42:33.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:42:33.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:42:33.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:42:33.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:42:33.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:42:33.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:42:33.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:42:33.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:42:33.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:42:33.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-09T13:42:33.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:42:33.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-09T13:42:34.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:42:34.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:42:34.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:42:34.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:42:34.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:42:34.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:42:34.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:42:34.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:42:34.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:42:34.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:42:34.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:42:34.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:42:34.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T13:42:34.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:42:34.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:42:34.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:243: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2024-03-09T13:42:34.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:244: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2024-03-09T13:42:34.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:42:34.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:42:34.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:42:34.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:42:34.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:42:34.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:42:34.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:42:34.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:42:34.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:42:34.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:42:34.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:42:34.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:42:34.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:42:34.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:42:34.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:42:34.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:42:34.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:42:34.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:42:34.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:42:34.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:42:34.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:42:34.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:42:34.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:42:34.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:42:34.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:42:34.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:42:34.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:42:35.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:42:35.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:42:35.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:42:35.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:42:35.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:42:35.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:42:35.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:42:35.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:42:35.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:42:35.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:42:35.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:42:35.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:42:35.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:42:35.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:42:35.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:42:35.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:42:35.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:42:35.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:42:35.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:42:35.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:42:35.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:42:35.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:42:35.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:42:35.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:42:35.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:42:35.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:42:35.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:42:35.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:42:35.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:42:35.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:42:35.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:42:35.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:42:35.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:42:35.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:42:35.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:42:35.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:42:35.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:42:35.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:42:35.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:42:35.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:42:35.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:42:35.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:42:35.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:42:35.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2024-03-09T13:42:35.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5746: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2024-03-09T13:42:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-03-09T13:42:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=false 2024-03-09T13:42:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-03-09T13:42:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-03-09T13:42:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-03-09T13:42:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:42:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:42:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:42:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:42:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:42:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T13:42:35.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:42:35.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:42:35.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:42:35.482 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:42:35.482 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:42:35.483 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:42:35.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:42:35.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T13:42:35.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:42:35.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:42:35.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:42:35.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:42:35.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:42:35.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:42:35.659 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:42:35.659 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:42:35.659 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:42:35.660 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:42:35.660 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:42:35.660 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:42:35.660 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:42:35.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:42:35.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:42:35.744 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:42:35.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:42:35.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:42:35.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:42:35.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:42:35.745 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:42:35.745 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:42:35.745 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:42:35.746 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:42:35.746 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:42:35.746 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:42:35.746 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:42:35.746 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:42:35.746 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:42:35.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-03-09T13:42:35.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:42:35.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:42:35.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:42:35.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:42:35.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:42:35.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:42:36.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:42:36.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:42:36.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:42:36.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:42:36.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:42:36.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:42:36.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:42:36.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:42:36.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:42:36.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-03-09T13:42:36.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-09T13:42:36.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-03-09T13:42:36.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:42:36.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:42:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:42:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:42:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:42:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:42:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:42:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:42:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:42:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:42:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:42:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:42:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:42:36.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:42:36.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:42:36.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:42:36.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:42:36.681 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:42:36.681 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:42:36.681 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:42:36.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:42:36.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:42:36.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:42:36.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:42:36.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:42:36.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:42:36.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:42:36.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:42:36.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:42:36.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:42:36.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:42:36.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:42:36.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:42:36.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:42:36.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:42:36.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=37480e6a-5ac8-4ec7-9083-5bae93ea29e1 2024-03-09T13:42:36.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 37480e6a-5ac8-4ec7-9083-5bae93ea29e1' 2024-03-09T13:42:36.683 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 37480e6a-5ac8-4ec7-9083-5bae93ea29e1 2024-03-09T13:42:36.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:42:36.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBMZ+xlEmZaKRAARxVMr9MSLnHQONMiDlL8JQ== 2024-03-09T13:42:36.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBMZ+xlEmZaKRAARxVMr9MSLnHQONMiDlL8JQ=="}' 2024-03-09T13:42:36.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 37480e6a-5ac8-4ec7-9083-5bae93ea29e1 -i td/osd-scrub-repair/0/new.json 2024-03-09T13:42:36.898 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:42:36.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:42:36.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMZ+xlEmZaKRAARxVMr9MSLnHQONMiDlL8JQ== --osd-uuid 37480e6a-5ac8-4ec7-9083-5bae93ea29e1 2024-03-09T13:42:36.939 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:36.939+0000 7fe3f9e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:42:36.950 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:36.949+0000 7fe3f9e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:42:36.951 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:36.951+0000 7fe3f9e0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:42:36.951 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:36.951+0000 7fe3f9e0d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:42:36.951 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:36.951+0000 7fe3f9e0d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:42:36.951 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:36.951+0000 7fe3f9e0d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:42:37.041 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:37.041+0000 7fe3f9e0d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:42:42.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:42:42.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:42:42.656 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:42:42.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:42:42.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:42:43.022 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:42:43.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:42:43.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:42:43.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:42:43.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:42:43.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:42:43.048 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:43.047+0000 7f55ca80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:42:43.050 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:43.049+0000 7f55ca80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:42:43.051 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:43.050+0000 7f55ca80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:42:43.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:42:43.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:42:43.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:42:43.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:42:43.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:42:43.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:43.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:42:43.296 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:42:43.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:43.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:42:43.571 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:43.569+0000 7f55ca80d640 -1 Falling back to public interface 2024-03-09T13:42:43.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:44.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:44.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:44.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:42:44.573 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:42:44.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:44.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:42:44.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:45.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:45.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:45.857 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:42:45.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:42:45.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:45.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:42:45.913 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:42:45.912+0000 7f55ca80d640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:42:46.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:47.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:47.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:47.181 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:42:47.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:42:47.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:47.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:42:47.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:48.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:48.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:48.510 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:42:48.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:42:48.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:48.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:42:48.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:49.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:49.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:49.915 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:42:49.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:42:49.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:49.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:42:50.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:51.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:51.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:51.772 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:42:51.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:42:51.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:51.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:42:51.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:52.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:52.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:52.489 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:42:52.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:42:52.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:52.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:42:52.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:53.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:53.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:53.872 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:42:53.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:42:53.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:53.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:42:54.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:55.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:55.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:55.157 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:42:55.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:42:55.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:55.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:42:55.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:56.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:56.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:56.549 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:42:56.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:42:56.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:56.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:42:56.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:57.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:57.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:57.912 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:42:57.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:42:57.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:57.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:42:58.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:42:59.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:42:59.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:42:59.424 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:42:59.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:42:59.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:42:59.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:42:59.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:00.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:00.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:00.897 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:43:00.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:43:00.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:00.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:43:01.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:02.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:02.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:02.356 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:43:02.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:43:02.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:02.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:43:02.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:03.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:03.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:03.931 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:43:03.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:43:03.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:03.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:43:04.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:05.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:05.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:05.434 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:43:05.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:43:05.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:05.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:43:06.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:07.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:07.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:07.011 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:43:07.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:43:07.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:07.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:43:07.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:08.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:08.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:08.449 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:43:08.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:43:08.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:08.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:43:08.798 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/517417208,v1:127.0.0.1:6803/517417208] [v2:127.0.0.1:6804/517417208,v1:127.0.0.1:6805/517417208] exists,up 37480e6a-5ac8-4ec7-9083-5bae93ea29e1 2024-03-09T13:43:08.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:43:08.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:43:08.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:43:08.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-09T13:43:08.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-03-09T13:43:08.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:43:08.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:43:08.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:43:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:43:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:43:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:43:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:43:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:43:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:43:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:43:08.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:43:08.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:43:08.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:43:08.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:43:08.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:43:08.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:43:08.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:43:08.810 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:43:08.810 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:43:08.810 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:43:08.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:43:08.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:43:08.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:43:08.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:43:08.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:43:08.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:43:08.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:43:08.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:43:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:43:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:43:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:43:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:43:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:43:08.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:43:08.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:43:08.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c59f0c6c-4dcd-47c5-9014-192965a74d4c 2024-03-09T13:43:08.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c59f0c6c-4dcd-47c5-9014-192965a74d4c' 2024-03-09T13:43:08.813 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 c59f0c6c-4dcd-47c5-9014-192965a74d4c 2024-03-09T13:43:08.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:43:08.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBsZ+xle8PNMBAAe/M6DK6IjoaYvAQWpuHIQg== 2024-03-09T13:43:08.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBsZ+xle8PNMBAAe/M6DK6IjoaYvAQWpuHIQg=="}' 2024-03-09T13:43:08.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c59f0c6c-4dcd-47c5-9014-192965a74d4c -i td/osd-scrub-repair/1/new.json 2024-03-09T13:43:09.221 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:43:09.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:43:09.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBsZ+xle8PNMBAAe/M6DK6IjoaYvAQWpuHIQg== --osd-uuid c59f0c6c-4dcd-47c5-9014-192965a74d4c 2024-03-09T13:43:09.261 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:09.261+0000 7f9fabe0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:43:09.306 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:09.306+0000 7f9fabe0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:43:09.307 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:09.307+0000 7f9fabe0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:43:09.308 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:09.307+0000 7f9fabe0d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:43:09.308 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:09.307+0000 7f9fabe0d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:43:09.308 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:09.307+0000 7f9fabe0d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:43:09.382 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:09.381+0000 7f9fabe0d640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:43:15.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:43:15.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:43:15.488 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:43:15.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:43:15.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:43:15.843 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:43:15.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:43:15.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:43:15.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:43:15.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:43:15.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:43:15.869 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:15.868+0000 7f249b013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:43:15.871 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:15.871+0000 7f249b013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:43:15.872 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:15.872+0000 7f249b013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:43:16.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:43:16.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:43:16.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:43:16.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:43:16.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:43:16.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:16.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:43:16.121 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:43:16.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:16.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:16.387 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:16.387+0000 7f249b013640 -1 Falling back to public interface 2024-03-09T13:43:16.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:17.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:17.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:17.395 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:43:17.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:43:17.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:17.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:17.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:18.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:18.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:18.676 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:43:18.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:43:18.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:18.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:18.751 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:18.751+0000 7f249b013640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:43:18.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:19.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:19.966 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:43:19.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:19.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:43:19.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:19.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:20.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:21.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:21.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:21.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:43:21.250 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:43:21.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:21.250 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:21.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:22.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:22.602 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:43:22.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:22.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:43:22.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:22.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:22.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:23.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:23.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:23.888 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:43:23.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:43:23.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:23.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:24.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:25.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:25.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:25.170 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:43:25.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:43:25.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:25.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:25.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:26.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:26.480 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:43:26.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:26.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:43:26.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:26.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:26.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:27.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:27.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:27.849 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:43:27.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:43:27.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:27.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:28.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:29.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:29.132 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:43:29.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:29.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:43:29.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:29.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:29.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:30.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:30.478 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:43:30.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:30.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:43:30.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:30.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:30.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:31.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:31.819 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:43:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:43:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:31.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:32.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:33.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:33.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:33.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:43:33.382 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:43:33.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:33.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:33.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:34.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:34.780 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:43:34.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:34.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:43:34.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:34.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:35.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:36.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:36.169 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:43:36.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:36.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:43:36.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:36.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:36.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:37.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:37.921 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:43:37.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:37.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:43:37.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:37.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:38.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:39.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:39.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:39.466 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:43:39.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:43:39.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:39.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:39.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:40.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:40.796 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:43:40.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:40.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:43:40.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:40.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:41.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:42.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:42.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:42.682 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:43:42.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:43:42.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:42.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:43:42.965 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/743280045,v1:127.0.0.1:6811/743280045] [v2:127.0.0.1:6812/743280045,v1:127.0.0.1:6813/743280045] exists,up c59f0c6c-4dcd-47c5-9014-192965a74d4c 2024-03-09T13:43:42.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:43:42.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:43:42.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:43:42.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-09T13:43:42.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-03-09T13:43:42.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:43:42.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:43:42.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T13:43:42.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:43:42.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:43:42.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:43:42.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:43:42.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:43:42.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:43:42.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:43:42.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:43:42.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:43:42.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:43:42.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:43:42.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:43:42.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:43:42.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:43:42.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:43:42.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:43:42.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:43:42.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:43:42.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:43:42.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1a349971-db7c-4aee-8f08-d0f53834dd84 2024-03-09T13:43:42.973 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 1a349971-db7c-4aee-8f08-d0f53834dd84 2024-03-09T13:43:42.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1a349971-db7c-4aee-8f08-d0f53834dd84' 2024-03-09T13:43:42.974 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:43:42.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCOZ+xlQxjPOhAATg7mv7QUUOTUfn3zmF326A== 2024-03-09T13:43:42.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCOZ+xlQxjPOhAATg7mv7QUUOTUfn3zmF326A=="}' 2024-03-09T13:43:42.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1a349971-db7c-4aee-8f08-d0f53834dd84 -i td/osd-scrub-repair/2/new.json 2024-03-09T13:43:43.315 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:43:43.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T13:43:43.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCOZ+xlQxjPOhAATg7mv7QUUOTUfn3zmF326A== --osd-uuid 1a349971-db7c-4aee-8f08-d0f53834dd84 2024-03-09T13:43:43.355 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:43.354+0000 7fce6060f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:43:43.400 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:43.398+0000 7fce6060f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:43:43.401 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:43.400+0000 7fce6060f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:43:43.402 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:43.400+0000 7fce6060f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:43:43.402 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:43.400+0000 7fce6060f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:43:43.402 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:43.400+0000 7fce6060f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:43:43.491 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:43.490+0000 7fce6060f640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T13:43:49.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T13:43:49.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:43:49.776 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T13:43:49.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T13:43:49.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:43:50.132 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T13:43:50.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T13:43:50.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:43:50.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:43:50.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:43:50.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:43:50.158 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:50.157+0000 7f62dd00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:43:50.160 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:50.159+0000 7f62dd00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:43:50.161 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:50.160+0000 7f62dd00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:43:50.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T13:43:50.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:43:50.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:43:50.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:43:50.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:43:50.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:50.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:43:50.407 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:43:50.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:50.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:43:50.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:50.679 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:50.678+0000 7f62dd00f640 -1 Falling back to public interface 2024-03-09T13:43:51.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:51.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:51.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:43:51.681 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:43:51.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:51.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:43:51.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:52.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:52.962 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:43:52.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:52.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:43:52.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:52.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:43:53.147 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:43:53.146+0000 7f62dd00f640 -1 osd.2 0 log_to_monitors true 2024-03-09T13:43:53.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:54.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:54.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:54.239 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:43:54.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:43:54.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:54.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:43:54.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:55.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:55.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:55.526 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:43:55.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:43:55.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:55.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:43:55.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:56.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:56.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:56.814 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:43:56.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:43:56.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:56.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:43:57.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:58.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:58.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:58.276 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:43:58.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:43:58.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:58.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:43:58.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:43:59.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:43:59.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:43:59.561 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:43:59.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:43:59.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:43:59.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:43:59.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:00.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:00.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:00.846 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:44:00.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:44:00.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:00.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:44:01.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:02.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:02.222 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:44:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:44:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:02.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:44:02.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:03.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:03.509 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:44:03.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:03.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:44:03.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:03.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:44:03.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:04.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:04.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:04.806 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:44:04.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:44:04.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:04.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:44:05.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:06.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:06.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:06.139 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:44:06.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:44:06.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:06.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:44:06.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:07.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:07.720 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:44:07.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:07.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:44:07.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:07.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:44:08.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:09.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:09.407 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:44:09.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:09.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:44:09.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:09.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:44:09.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:10.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:10.752 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:44:10.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:10.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:44:10.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:10.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:44:11.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:12.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:12.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:12.617 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:44:12.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:44:12.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:12.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:44:12.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:13.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:13.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:13.994 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:44:13.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:44:13.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:13.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:44:14.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:15.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:15.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:15.432 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:44:15.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:44:15.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:15.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:44:15.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:16.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:16.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:16.849 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:44:16.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:44:16.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:16.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:44:17.126 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/674664186,v1:127.0.0.1:6819/674664186] [v2:127.0.0.1:6820/674664186,v1:127.0.0.1:6821/674664186] exists,up 1a349971-db7c-4aee-8f08-d0f53834dd84 2024-03-09T13:44:17.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:44:17.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:44:17.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:44:17.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-03-09T13:44:17.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T13:44:17.395 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T13:44:17.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T13:44:17.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T13:44:17.817 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T13:44:17.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:44:18.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T13:44:21.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-03-09T13:44:21.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-09T13:44:21.554 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'foo' created 2024-03-09T13:44:21.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:44:22.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2024-03-09T13:44:22.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T13:44:22.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T13:44:22.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-09T13:44:22.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T13:44:22.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-03-09T13:44:23.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T13:44:23.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T13:44:24.049 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' created 2024-03-09T13:44:24.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:44:25.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-09T13:44:25.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T13:44:25.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:44:25.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:44:25.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:44:25.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:44:25.063 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:44:25.063 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:44:25.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:44:25.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:44:25.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:44:25.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:44:25.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:44:25.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:44:25.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:44:25.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:44:25.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:44:25.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:44:25.438 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:44:25.438 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:44:25.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:44:25.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:44:25.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:44:25.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-03-09T13:44:25.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-03-09T13:44:25.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-03-09T13:44:25.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:44:25.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:44:25.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-03-09T13:44:25.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-03-09T13:44:25.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672973' 2024-03-09T13:44:25.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:44:25.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:44:25.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-03-09T13:44:25.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-03-09T13:44:25.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672973 2-64424509446' 2024-03-09T13:44:25.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:44:25.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-03-09T13:44:25.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:44:25.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:44:25.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-03-09T13:44:25.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:44:25.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-03-09T13:44:25.818 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836502 2024-03-09T13:44:25.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-03-09T13:44:25.819 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:44:26.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-03-09T13:44:26.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:44:27.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:44:27.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:44:27.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-03-09T13:44:27.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:44:27.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-03-09T13:44:27.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:44:27.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:44:27.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-03-09T13:44:27.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:44:27.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-03-09T13:44:27.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-03-09T13:44:27.411 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672973 2024-03-09T13:44:27.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:44:27.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-03-09T13:44:27.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:44:27.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-03-09T13:44:27.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:44:27.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:44:27.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-03-09T13:44:27.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:44:27.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-03-09T13:44:27.689 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509446 2024-03-09T13:44:27.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-03-09T13:44:27.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:44:27.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509446 2024-03-09T13:44:27.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:44:27.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:44:27.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:44:28.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:44:28.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:44:28.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:44:28.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:44:28.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:44:28.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:44:28.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:44:28.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:44:28.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:44:28.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:44:28.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:44:28.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:44:28.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-09T13:44:28.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-03-09T13:44:28.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:44:28.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-09T13:44:28.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:44:28.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:44:29.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:44:29.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:44:29.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:44:29.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:44:29.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:44:29.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:44:29.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:44:29.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:44:29.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:44:29.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:44:29.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:44:29.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:44:29.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:44:29.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:44:29.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:44:29.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T13:44:29.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-03-09T13:44:29.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:44:29.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:44:29.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:44:29.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:44:29.631 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:44:29.631 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:44:29.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:44:29.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:44:29.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:44:29.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:44:29.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:44:29.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:44:29.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:44:29.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:44:29.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:44:29.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:44:29.987 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:44:29.987 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:44:29.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:44:29.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:44:29.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:44:30.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836504 2024-03-09T13:44:30.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836504 2024-03-09T13:44:30.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504' 2024-03-09T13:44:30.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:44:30.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:44:30.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-09T13:44:30.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-09T13:44:30.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672975' 2024-03-09T13:44:30.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:44:30.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:44:30.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509448 2024-03-09T13:44:30.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509448 2024-03-09T13:44:30.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836504 1-42949672975 2-64424509448' 2024-03-09T13:44:30.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:44:30.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836504 2024-03-09T13:44:30.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:44:30.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:44:30.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836504 2024-03-09T13:44:30.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:44:30.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836504 2024-03-09T13:44:30.331 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836504 2024-03-09T13:44:30.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-03-09T13:44:30.331 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:44:30.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836504 2024-03-09T13:44:30.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:44:31.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:44:31.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:44:31.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836504 2024-03-09T13:44:31.885 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:44:31.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-09T13:44:31.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:44:31.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:44:31.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-09T13:44:31.886 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:44:31.887 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672975 2024-03-09T13:44:31.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-09T13:44:31.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-09T13:44:31.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:44:32.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-09T13:44:32.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:44:32.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509448 2024-03-09T13:44:32.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:44:32.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:44:32.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509448 2024-03-09T13:44:32.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:44:32.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509448 2024-03-09T13:44:32.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-03-09T13:44:32.185 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509448 2024-03-09T13:44:32.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:44:32.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509448 -lt 64424509448 2024-03-09T13:44:32.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:44:32.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:44:32.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:44:32.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:44:32.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:44:32.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:44:32.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:44:32.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:44:32.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:44:32.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:44:32.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:44:33.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:44:33.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:44:33.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:44:33.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:44:33.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:44:33.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:44:33.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:44:33.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-03-09T13:44:33.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:44:33.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-03-09T13:44:33.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-03-09T13:44:33.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:44:33.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:44:33.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-03-09T13:44:33.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:44:33.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:44:33.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:44:33.780 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:44:33.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:44:34.243 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:44:34.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:44:34.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:44:34.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:44:34.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-03-09T13:44:34.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T13:44:34.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:44:34.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-03-09T13:44:34.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-03-09T13:44:34.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:44:34.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:44:34.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:44:34.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:44:34.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:44:34.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:44:34.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:44:34.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:44:34.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:44:34.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:44:34.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:44:34.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:44:34.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:44:34.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:44:34.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:44:34.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:44:34.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:44:34.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:44:34.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:44:34.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:44:34.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:44:34.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:44:34.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:44:34.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:44:36.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:44:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:44:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:44:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:44:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:44:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:44:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:44:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:44:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:44:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:44:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:44:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:44:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:44:36.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:44:36.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:44:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:44:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:44:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:44:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:44:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:44:36.865 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:44:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:44:36.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:44:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:44:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:44:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:44:36.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:44:36.892 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:44:36.891+0000 7f5c39d93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:44:36.894 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:44:36.893+0000 7f5c39d93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:44:36.895 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:44:36.894+0000 7f5c39d93640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:44:37.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:44:37.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:44:37.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:44:37.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:44:37.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:44:37.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:37.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:44:37.147 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:44:37.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:37.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:44:37.415 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:44:37.414+0000 7f5c39d93640 -1 Falling back to public interface 2024-03-09T13:44:37.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:38.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:38.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:38.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:44:38.423 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:44:38.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:38.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:44:38.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:39.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:39.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:39.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:44:39.697 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:44:39.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:39.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:44:39.781 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:44:39.780+0000 7f5c39d93640 -1 osd.1 36 log_to_monitors true 2024-03-09T13:44:39.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:40.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:40.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:40.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:44:40.997 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:44:40.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:40.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:44:41.303 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/545946558,v1:127.0.0.1:6811/545946558] [v2:127.0.0.1:6812/545946558,v1:127.0.0.1:6813/545946558] exists,up c59f0c6c-4dcd-47c5-9014-192965a74d4c 2024-03-09T13:44:41.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:44:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:44:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:44:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:44:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:44:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:44:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:44:41.304 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:44:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:44:41.304 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:44:41.304 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:44:41.304 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:44:41.304 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:44:41.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:44:41.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:44:41.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:44:41.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:44:41.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:44:41.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:44:41.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:44:41.772 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:44:41.772 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:44:41.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:44:41.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:44:41.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:44:41.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836507 2024-03-09T13:44:41.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836507 2024-03-09T13:44:41.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507' 2024-03-09T13:44:41.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:44:41.876 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:44:41.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-09T13:44:41.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-09T13:44:41.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-171798691842' 2024-03-09T13:44:41.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:44:41.980 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:44:42.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509452 2024-03-09T13:44:42.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509452 2024-03-09T13:44:42.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836507 1-171798691842 2-64424509452' 2024-03-09T13:44:42.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:44:42.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836507 2024-03-09T13:44:42.084 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:44:42.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:44:42.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836507 2024-03-09T13:44:42.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:44:42.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836507 2024-03-09T13:44:42.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-03-09T13:44:42.087 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836507 2024-03-09T13:44:42.087 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:44:42.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836507 2024-03-09T13:44:42.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:44:43.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:44:43.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:44:43.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836507 -lt 21474836507 2024-03-09T13:44:43.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:44:43.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-03-09T13:44:43.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:44:43.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:44:43.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-03-09T13:44:43.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:44:43.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-09T13:44:43.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-03-09T13:44:43.662 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 171798691842 2024-03-09T13:44:43.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:44:43.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-09T13:44:43.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:44:43.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509452 2024-03-09T13:44:43.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:44:43.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:44:43.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509452 2024-03-09T13:44:43.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:44:43.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509452 2024-03-09T13:44:43.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-03-09T13:44:43.936 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509452 2024-03-09T13:44:43.937 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:44:44.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509452 -lt 64424509452 2024-03-09T13:44:44.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:44:44.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:44:44.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:44:44.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:44:44.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:44:44.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:44:44.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:44:44.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:44:44.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:44:44.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:44:44.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:44:44.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:44:44.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:44:44.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:44:44.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:44:45.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:44:45.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:44:45.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:44:45.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:44:45.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-03-09T13:44:45.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-03-09T13:44:45.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:44:45.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:44:45.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-03-09T13:44:45.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:44:45.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:44:45.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:44:45.539 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:44:45.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:44:45.931 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:44:45.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:44:45.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:44:45.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:44:46.168 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-03-09T13:44:46.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-09T13:44:46.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:44:46.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-03-09T13:44:46.171 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-03-09T13:44:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-03-09T13:44:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.2186e-05 s, 32.9 MB/s 2024-03-09T13:44:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:44:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:44:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:44:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:44:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:44:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:44:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:44:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:44:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:44:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:44:46.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:44:46.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:44:46.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:44:46.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:44:46.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:44:46.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:44:47.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:44:47.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:44:47.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:44:47.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:44:47.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:44:47.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:44:47.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:44:47.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:44:49.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:44:49.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:44:49.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:44:49.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:44:49.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:44:49.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:44:49.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:44:49.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:44:49.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:44:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:44:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:44:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:44:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:44:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:44:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:44:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:44:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:44:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:44:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:44:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:44:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:44:49.272 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:44:49.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:44:49.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:44:49.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:44:49.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:44:49.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:44:49.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:44:49.299 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:44:49.299+0000 7f09dec13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:44:49.302 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:44:49.302+0000 7f09dec13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:44:49.303 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:44:49.303+0000 7f09dec13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:44:49.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:44:49.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:44:49.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:44:49.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:44:49.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:44:49.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:49.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:44:49.555 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:44:49.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:49.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:44:49.822 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:44:49.822+0000 7f09dec13640 -1 Falling back to public interface 2024-03-09T13:44:49.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:50.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:50.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:50.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:44:50.826 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:44:50.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:50.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:44:51.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:44:52.099 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:44:52.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:52.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:44:52.145 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:44:52.144+0000 7f09dec13640 -1 osd.0 43 log_to_monitors true 2024-03-09T13:44:52.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:44:53.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:44:53.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:44:53.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:44:53.386 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:44:53.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:44:53.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:44:53.661 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 48 up_thru 48 down_at 44 last_clean_interval [5,43) [v2:127.0.0.1:6802/3991300077,v1:127.0.0.1:6803/3991300077] [v2:127.0.0.1:6804/3991300077,v1:127.0.0.1:6805/3991300077] exists,up 37480e6a-5ac8-4ec7-9083-5bae93ea29e1 2024-03-09T13:44:53.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:44:53.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:44:53.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:44:53.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:44:53.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:44:53.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:44:53.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:44:53.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:44:53.662 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:44:53.662 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:44:53.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:44:53.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:44:53.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:44:53.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:44:53.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:44:53.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:44:53.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:44:53.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:44:53.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:44:54.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:44:54.193 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:44:54.193 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:44:54.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:44:54.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:44:54.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:44:54.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430210 2024-03-09T13:44:54.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430210 2024-03-09T13:44:54.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210' 2024-03-09T13:44:54.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:44:54.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:44:54.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-03-09T13:44:54.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-03-09T13:44:54.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-171798691845' 2024-03-09T13:44:54.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:44:54.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:44:54.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-09T13:44:54.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-09T13:44:54.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430210 1-171798691845 2-64424509455' 2024-03-09T13:44:54.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:44:54.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430210 2024-03-09T13:44:54.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:44:54.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:44:54.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430210 2024-03-09T13:44:54.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:44:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430210 2024-03-09T13:44:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430210' 2024-03-09T13:44:54.541 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 206158430210 2024-03-09T13:44:54.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:44:54.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 206158430210 2024-03-09T13:44:54.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:44:55.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:44:55.817 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:44:56.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430210 -lt 206158430210 2024-03-09T13:44:56.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:44:56.102 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691845 2024-03-09T13:44:56.102 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:44:56.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:44:56.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691845 2024-03-09T13:44:56.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:44:56.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-03-09T13:44:56.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-03-09T13:44:56.104 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 171798691845 2024-03-09T13:44:56.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:44:56.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691845 2024-03-09T13:44:56.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:44:56.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-09T13:44:56.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:44:56.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:44:56.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-09T13:44:56.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:44:56.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-09T13:44:56.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-09T13:44:56.379 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509455 2024-03-09T13:44:56.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:44:56.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-09T13:44:56.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:44:56.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:44:56.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:44:56.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:44:56.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:44:56.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:44:56.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:44:56.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:44:56.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:44:56.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:44:56.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:44:57.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:44:57.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:44:57.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:44:57.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:44:57.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:44:57.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:44:57.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:44:57.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:44:57.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-03-09T13:44:57.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-03-09T13:44:57.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:44:57.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:44:57.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-03-09T13:44:57.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:44:57.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:44:57.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:44:58.119 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:44:58.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:44:58.496 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:44:58.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:44:58.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:44:58.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:44:58.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-03-09T13:44:58.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T13:44:58.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:44:58.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-03-09T13:44:58.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:44:58.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:44:58.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:44:58.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:44:58.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-03-09T13:44:58.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:44:58.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:44:58.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:44:58.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:44:58.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:44:58.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:44:58.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:44:58.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:44:58.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:44:58.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:44:59.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:44:59.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-03-09T13:44:59.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:44:59.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:44:59.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:44:59.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:44:59.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:44:59.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-03-09T13:45:00.834 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-03-09T13:45:01.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:45:01.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:45:01.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:45:01.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:45:01.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:45:01.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:45:01.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:45:01.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:45:01.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:45:01.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:45:01.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:45:01.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:45:01.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:45:01.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:45:01.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:45:01.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:45:01.353 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:45:01.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:45:01.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:45:01.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:45:01.354 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:45:01.354 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:45:01.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:45:01.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:45:01.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:45:01.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:45:01.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:45:01.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:45:01.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:45:01.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:45:01.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:45:01.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:45:01.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:45:01.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:45:01.355 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:45:01.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:45:01.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:45:01.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:45:01.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:45:01.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:45:01.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:45:01.382 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:01.381+0000 7fe26b926640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:01.384 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:01.384+0000 7fe26b926640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:01.385 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:01.385+0000 7fe26b926640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:01.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:45:01.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:45:01.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:45:01.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:45:01.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:45:01.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:01.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:45:01.639 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:45:01.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:01.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:45:01.901 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:01.901+0000 7fe26b926640 -1 Falling back to public interface 2024-03-09T13:45:01.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:02.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:02.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:02.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:45:02.914 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:45:02.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:02.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:45:03.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:04.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:04.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:04.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:45:04.197 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:45:04.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:04.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:45:04.230 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:04.230+0000 7fe26b926640 -1 osd.1 52 log_to_monitors true 2024-03-09T13:45:04.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:05.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:05.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:05.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:45:05.488 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:45:05.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:05.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:45:05.928 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 53 last_clean_interval [40,52) [v2:127.0.0.1:6810/3624691923,v1:127.0.0.1:6811/3624691923] [v2:127.0.0.1:6812/3624691923,v1:127.0.0.1:6813/3624691923] exists,up c59f0c6c-4dcd-47c5-9014-192965a74d4c 2024-03-09T13:45:05.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:45:05.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:45:05.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:45:05.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:45:05.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:45:05.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:45:05.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:45:05.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:45:05.928 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:45:05.928 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:45:05.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:45:05.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:45:05.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:45:06.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:45:06.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:45:06.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:45:06.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:45:06.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:45:06.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:45:06.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:45:06.421 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:45:06.421 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:45:06.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:45:06.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:06.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:45:06.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430213 2024-03-09T13:45:06.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430213 2024-03-09T13:45:06.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213' 2024-03-09T13:45:06.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:06.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:45:06.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135874 2024-03-09T13:45:06.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135874 2024-03-09T13:45:06.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213 1-244813135874' 2024-03-09T13:45:06.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:06.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:45:06.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509459 2024-03-09T13:45:06.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509459 2024-03-09T13:45:06.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-206158430213 1-244813135874 2-64424509459' 2024-03-09T13:45:06.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:06.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-206158430213 2024-03-09T13:45:06.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:06.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:45:06.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-206158430213 2024-03-09T13:45:06.733 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:06.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430213 2024-03-09T13:45:06.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 206158430213' 2024-03-09T13:45:06.734 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 206158430213 2024-03-09T13:45:06.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:45:07.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430213 -lt 206158430213 2024-03-09T13:45:07.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:07.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135874 2024-03-09T13:45:07.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:07.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:45:07.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135874 2024-03-09T13:45:07.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:07.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135874 2024-03-09T13:45:07.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135874' 2024-03-09T13:45:07.067 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 244813135874 2024-03-09T13:45:07.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:45:07.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135874 -lt 244813135874 2024-03-09T13:45:07.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:07.342 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509459 2024-03-09T13:45:07.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:07.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:45:07.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509459 2024-03-09T13:45:07.344 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:07.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509459 2024-03-09T13:45:07.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-03-09T13:45:07.345 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509459 2024-03-09T13:45:07.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:45:07.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509459 2024-03-09T13:45:07.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:45:08.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:45:08.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:45:08.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509459 -lt 64424509459 2024-03-09T13:45:08.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:45:08.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:45:08.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:45:09.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:45:09.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:45:09.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:45:09.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:45:09.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:45:09.251 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:45:09.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:45:09.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:45:09.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:45:09.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:45:09.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:45:09.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:45:09.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:45:09.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:45:09.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:45:09.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:45:09.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-03-09T13:45:09.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-03-09T13:45:09.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:45:09.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:45:09.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-03-09T13:45:09.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:45:09.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:45:09.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:45:10.230 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:45:10.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:45:10.748 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:45:10.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:45:10.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:45:10.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:45:11.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-03-09T13:45:11.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-09T13:45:11.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:45:11.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-03-09T13:45:11.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-03-09T13:45:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-03-09T13:45:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T13:45:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:45:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:45:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:45:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:45:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T13:45:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:45:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:45:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:45:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:45:11.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:45:11.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:45:11.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:45:11.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:45:11.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:45:11.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:45:11.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:45:11.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T13:45:11.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:45:11.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:45:11.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:45:11.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:45:11.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:45:11.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T13:45:13.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:45:13.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:45:13.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:45:13.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:45:13.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:45:13.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:45:13.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:45:13.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:45:13.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:45:13.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:45:13.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:45:13.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:45:13.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:45:13.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:45:13.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:45:13.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:45:13.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:45:13.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:45:13.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:45:13.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:45:13.892 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:45:13.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:45:13.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:45:13.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:45:13.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:45:13.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:45:13.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:45:13.919 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:13.918+0000 7f6d7f919640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:13.921 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:13.920+0000 7f6d7f919640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:13.922 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:13.921+0000 7f6d7f919640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:14.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:45:14.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:45:14.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:45:14.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:45:14.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:45:14.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:14.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:45:14.175 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:45:14.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:14.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:45:14.445 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:14.444+0000 7f6d7f919640 -1 Falling back to public interface 2024-03-09T13:45:14.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:15.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:15.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:15.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:45:15.451 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:45:15.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:15.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:45:15.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:16.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:16.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:16.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:45:16.736 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:45:16.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:16.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:45:16.742 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:16.741+0000 7f6d7f919640 -1 osd.0 61 log_to_monitors true 2024-03-09T13:45:17.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:18.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:18.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:18.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:45:18.082 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:45:18.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:18.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:45:18.478 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 65 up_thru 65 down_at 62 last_clean_interval [48,61) [v2:127.0.0.1:6802/3937268082,v1:127.0.0.1:6803/3937268082] [v2:127.0.0.1:6804/3937268082,v1:127.0.0.1:6805/3937268082] exists,up 37480e6a-5ac8-4ec7-9083-5bae93ea29e1 2024-03-09T13:45:18.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:45:18.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:45:18.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:45:18.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:45:18.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:45:18.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:45:18.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:45:18.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:45:18.479 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:45:18.479 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:45:18.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:45:18.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:45:18.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:45:18.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:45:18.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:45:18.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:45:18.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:45:18.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:45:18.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:45:18.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:45:18.972 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:45:18.972 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:45:18.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:45:18.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:18.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:45:19.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874242 2024-03-09T13:45:19.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874242 2024-03-09T13:45:19.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-279172874242' 2024-03-09T13:45:19.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:19.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:45:19.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135877 2024-03-09T13:45:19.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135877 2024-03-09T13:45:19.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-279172874242 1-244813135877' 2024-03-09T13:45:19.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:19.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:45:19.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509462 2024-03-09T13:45:19.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509462 2024-03-09T13:45:19.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-279172874242 1-244813135877 2-64424509462' 2024-03-09T13:45:19.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:19.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-279172874242 2024-03-09T13:45:19.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:19.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:45:19.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-279172874242 2024-03-09T13:45:19.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:19.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874242 2024-03-09T13:45:19.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 279172874242' 2024-03-09T13:45:19.431 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 279172874242 2024-03-09T13:45:19.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:45:19.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 279172874242 2024-03-09T13:45:19.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:45:20.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:45:20.704 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:45:20.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874242 -lt 279172874242 2024-03-09T13:45:20.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:20.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135877 2024-03-09T13:45:20.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:20.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:45:20.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135877 2024-03-09T13:45:20.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:20.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135877 2024-03-09T13:45:20.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135877' 2024-03-09T13:45:20.994 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 244813135877 2024-03-09T13:45:20.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:45:21.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135878 -lt 244813135877 2024-03-09T13:45:21.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:21.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509462 2024-03-09T13:45:21.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:21.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:45:21.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509462 2024-03-09T13:45:21.266 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:21.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509462 2024-03-09T13:45:21.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-03-09T13:45:21.267 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509462 2024-03-09T13:45:21.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:45:21.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509462 2024-03-09T13:45:21.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:45:21.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:45:21.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:45:21.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:45:21.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:45:21.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:45:21.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:45:21.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:45:21.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:45:21.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:45:21.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:45:22.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:45:22.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:45:22.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:45:22.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:45:22.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:45:22.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:45:22.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:45:22.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:45:22.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:45:22.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:45:22.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:45:22.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T13:45:22.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:45:22.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:45:22.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T13:45:22.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:45:22.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:45:22.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T13:45:25.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T13:45:25.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:45:25.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:45:25.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T13:45:25.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:45:25.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:45:25.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:45:25.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:45:25.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:45:25.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:45:25.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:45:25.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:45:25.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:45:25.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:45:25.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:45:25.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:45:25.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:45:25.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:45:25.097 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:45:25.097 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:45:25.097 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:45:25.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:45:25.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:45:25.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:45:25.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:45:25.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:45:25.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:45:25.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:45:25.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:45:25.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:45:25.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:45:25.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:45:25.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:45:25.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T13:45:25.098 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-03-09T13:45:25.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:45:25.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T13:45:25.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T13:45:25.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:45:25.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:45:25.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:45:25.127 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:25.125+0000 7f9bb2321640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:25.129 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:25.128+0000 7f9bb2321640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:25.131 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:25.129+0000 7f9bb2321640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:25.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T13:45:25.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:45:25.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:45:25.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:45:25.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:45:25.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:25.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:45:25.382 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:45:25.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:25.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:45:25.652 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:25.651+0000 7f9bb2321640 -1 Falling back to public interface 2024-03-09T13:45:25.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:26.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:26.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:26.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:45:26.655 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:45:26.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:26.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:45:26.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:27.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:27.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:27.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:45:27.964 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:45:27.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:27.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:45:28.000 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:27.998+0000 7f9bb2321640 -1 osd.2 68 log_to_monitors true 2024-03-09T13:45:28.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:29.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:29.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:29.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:45:29.245 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:45:29.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:29.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:45:29.594 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 73 up_thru 73 down_at 69 last_clean_interval [15,68) [v2:127.0.0.1:6818/2387386835,v1:127.0.0.1:6819/2387386835] [v2:127.0.0.1:6820/2387386835,v1:127.0.0.1:6821/2387386835] exists,up 1a349971-db7c-4aee-8f08-d0f53834dd84 2024-03-09T13:45:29.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:45:29.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:45:29.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:45:29.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:45:29.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:45:29.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:45:29.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:45:29.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:45:29.595 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:45:29.595 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:45:29.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:45:29.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:45:29.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:45:29.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:45:29.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:45:29.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:45:29.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:45:29.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:45:29.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:45:30.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:45:30.051 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:45:30.051 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:45:30.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:45:30.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:30.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:45:30.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874245 2024-03-09T13:45:30.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874245 2024-03-09T13:45:30.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-279172874245' 2024-03-09T13:45:30.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:30.156 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:45:30.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135881 2024-03-09T13:45:30.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135881 2024-03-09T13:45:30.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-279172874245 1-244813135881' 2024-03-09T13:45:30.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:30.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:45:30.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612610 2024-03-09T13:45:30.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612610 2024-03-09T13:45:30.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-279172874245 1-244813135881 2-313532612610' 2024-03-09T13:45:30.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:30.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-279172874245 2024-03-09T13:45:30.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:30.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:45:30.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-279172874245 2024-03-09T13:45:30.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:30.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874245 2024-03-09T13:45:30.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 279172874245' 2024-03-09T13:45:30.366 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 279172874245 2024-03-09T13:45:30.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:45:30.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874244 -lt 279172874245 2024-03-09T13:45:30.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:45:31.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:45:31.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:45:31.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874245 -lt 279172874245 2024-03-09T13:45:31.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:31.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135881 2024-03-09T13:45:31.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:31.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:45:31.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135881 2024-03-09T13:45:31.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:31.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135881 2024-03-09T13:45:31.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135881' 2024-03-09T13:45:31.919 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 244813135881 2024-03-09T13:45:31.919 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:45:32.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135881 -lt 244813135881 2024-03-09T13:45:32.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:32.194 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612610 2024-03-09T13:45:32.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:32.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:45:32.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612610 2024-03-09T13:45:32.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:32.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612610 2024-03-09T13:45:32.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612610' 2024-03-09T13:45:32.197 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 313532612610 2024-03-09T13:45:32.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:45:32.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612610 -lt 313532612610 2024-03-09T13:45:32.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:45:32.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:45:32.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:45:32.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:45:32.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:45:32.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:45:32.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:45:32.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:45:32.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:45:32.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:45:32.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:45:33.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:45:33.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:45:33.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:45:33.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:45:33.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:45:33.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:45:33.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:45:33.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-03-09T13:45:33.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T13:45:33.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:45:33.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:45:33.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T13:45:33.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:45:33.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T13:45:33.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:45:33.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:45:33.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T13:45:33.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:45:33.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T13:45:33.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:45:33.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:45:33.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:45:33.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:45:33.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:45:33.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:45:33.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T13:45:33.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:45:33.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:45:33.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T13:45:33.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:45:33.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:45:33.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T13:45:35.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T13:45:35.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:45:35.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:45:35.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T13:45:35.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:45:35.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:45:35.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:45:35.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:45:35.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:45:35.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:45:35.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:45:35.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:45:35.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:45:35.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:45:35.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:45:35.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:45:35.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:45:35.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:45:35.945 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:45:35.945 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:45:35.945 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:45:35.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:45:35.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:45:35.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:45:35.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:45:35.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:45:35.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:45:35.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:45:35.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:45:35.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:45:35.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:45:35.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:45:35.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:45:35.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T13:45:35.946 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-03-09T13:45:35.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:45:35.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T13:45:35.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T13:45:35.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:45:35.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:45:35.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:45:35.973 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:35.973+0000 7f79f8415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:35.976 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:35.976+0000 7f79f8415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:35.977 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:35.977+0000 7f79f8415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:36.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T13:45:36.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:45:36.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:45:36.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:45:36.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:45:36.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:36.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:45:36.231 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:45:36.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:36.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:45:36.491 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:36.491+0000 7f79f8415640 -1 Falling back to public interface 2024-03-09T13:45:36.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:37.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:37.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:37.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:45:37.504 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:45:37.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:37.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:45:37.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:38.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:38.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:38.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:45:38.786 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:45:38.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:38.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:45:38.835 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:38.835+0000 7f79f8415640 -1 osd.2 75 log_to_monitors true 2024-03-09T13:45:39.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:40.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:40.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:40.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:45:40.122 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:45:40.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:40.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:45:40.519 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 80 up_thru 80 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/2177810815,v1:127.0.0.1:6819/2177810815] [v2:127.0.0.1:6820/2177810815,v1:127.0.0.1:6821/2177810815] exists,up 1a349971-db7c-4aee-8f08-d0f53834dd84 2024-03-09T13:45:40.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:45:40.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:45:40.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:45:40.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:45:40.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:45:40.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:45:40.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:45:40.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:45:40.520 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:45:40.520 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:45:40.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:45:40.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:45:40.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:45:40.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:45:40.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:45:40.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:45:40.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:45:40.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:45:40.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:45:40.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:45:40.981 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:45:40.981 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:45:40.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:45:40.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:40.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:45:41.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874248 2024-03-09T13:45:41.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874248 2024-03-09T13:45:41.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-279172874248' 2024-03-09T13:45:41.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:41.087 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:45:41.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=244813135884 2024-03-09T13:45:41.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 244813135884 2024-03-09T13:45:41.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-279172874248 1-244813135884' 2024-03-09T13:45:41.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:41.193 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:45:41.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383682 2024-03-09T13:45:41.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383682 2024-03-09T13:45:41.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-279172874248 1-244813135884 2-343597383682' 2024-03-09T13:45:41.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:41.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-279172874248 2024-03-09T13:45:41.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:41.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:45:41.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-279172874248 2024-03-09T13:45:41.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:41.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874248 2024-03-09T13:45:41.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 279172874248' 2024-03-09T13:45:41.299 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 279172874248 2024-03-09T13:45:41.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:45:41.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874247 -lt 279172874248 2024-03-09T13:45:41.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:45:42.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:45:42.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:45:42.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874249 -lt 279172874248 2024-03-09T13:45:42.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:42.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-244813135884 2024-03-09T13:45:42.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:42.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:45:42.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-244813135884 2024-03-09T13:45:42.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:42.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=244813135884 2024-03-09T13:45:42.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 244813135884' 2024-03-09T13:45:42.868 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 244813135884 2024-03-09T13:45:42.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:45:43.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 244813135884 -lt 244813135884 2024-03-09T13:45:43.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:43.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383682 2024-03-09T13:45:43.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:43.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:45:43.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383682 2024-03-09T13:45:43.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:43.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383682 2024-03-09T13:45:43.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383682' 2024-03-09T13:45:43.150 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 343597383682 2024-03-09T13:45:43.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:45:43.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383682 -lt 343597383682 2024-03-09T13:45:43.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:45:43.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:45:43.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:45:43.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:45:43.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:45:43.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:45:43.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:45:43.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:45:43.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:45:43.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:45:43.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:45:44.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:45:44.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:45:44.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:45:44.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:45:44.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:45:44.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:45:44.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:45:44.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-09T13:45:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:45:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-03-09T13:45:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-03-09T13:45:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:45:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:45:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-03-09T13:45:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:45:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:45:44.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:45:44.739 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:45:44.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:45:45.258 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:45:45.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:45:45.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:45:45.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:45:45.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-03-09T13:45:45.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T13:45:45.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:45:45.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-03-09T13:45:45.498 INFO:tasks.workunit.client.0.smithi162.stderr:2+0 records in 2024-03-09T13:45:45.498 INFO:tasks.workunit.client.0.smithi162.stderr:2+0 records out 2024-03-09T13:45:45.498 INFO:tasks.workunit.client.0.smithi162.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 7.7735e-05 s, 52.7 MB/s 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:45:45.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:45:45.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:45:45.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:45:45.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:45:45.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:45:45.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:45:45.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:45:45.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:45:45.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:45:45.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:45:45.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:45:48.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:45:48.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:45:48.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:45:48.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:45:48.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:45:48.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:45:48.049 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:45:48.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:45:48.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:45:48.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:45:48.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:45:48.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:45:48.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:45:48.072 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:48.072+0000 7f682ac13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:48.074 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:48.074+0000 7f682ac13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:48.075 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:48.075+0000 7f682ac13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:48.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:45:48.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:45:48.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:45:48.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:45:48.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:45:48.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:48.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:45:48.325 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:45:48.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:48.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:45:48.591 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:48.591+0000 7f682ac13640 -1 Falling back to public interface 2024-03-09T13:45:48.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:49.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:49.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:49.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:45:49.599 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:45:49.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:49.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:45:49.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:50.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:50.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:50.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:45:50.881 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:45:50.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:50.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:45:51.043 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:51.043+0000 7f682ac13640 -1 osd.1 84 log_to_monitors true 2024-03-09T13:45:51.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:45:52.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:45:52.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:45:52.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:45:52.173 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:45:52.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:45:52.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:45:52.533 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 85 last_clean_interval [57,84) [v2:127.0.0.1:6810/239783592,v1:127.0.0.1:6811/239783592] [v2:127.0.0.1:6812/239783592,v1:127.0.0.1:6813/239783592] exists,up c59f0c6c-4dcd-47c5-9014-192965a74d4c 2024-03-09T13:45:52.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:45:52.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:45:52.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:45:52.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:45:52.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:45:52.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:45:52.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:45:52.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:45:52.534 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:45:52.534 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:45:52.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:45:52.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:45:52.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:45:52.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:45:52.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:45:52.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:45:52.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:45:52.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:45:52.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:45:52.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:45:52.973 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:45:52.973 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:45:52.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:45:52.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:52.973 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:45:53.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=279172874252 2024-03-09T13:45:53.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 279172874252 2024-03-09T13:45:53.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-279172874252' 2024-03-09T13:45:53.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:53.079 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:45:53.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089346 2024-03-09T13:45:53.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089346 2024-03-09T13:45:53.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-279172874252 1-382252089346' 2024-03-09T13:45:53.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:45:53.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:45:53.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383685 2024-03-09T13:45:53.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383685 2024-03-09T13:45:53.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-279172874252 1-382252089346 2-343597383685' 2024-03-09T13:45:53.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:53.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-279172874252 2024-03-09T13:45:53.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:53.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:45:53.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-279172874252 2024-03-09T13:45:53.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:53.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=279172874252 2024-03-09T13:45:53.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 279172874252' 2024-03-09T13:45:53.289 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 279172874252 2024-03-09T13:45:53.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:45:53.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874251 -lt 279172874252 2024-03-09T13:45:53.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:45:54.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:45:54.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:45:54.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 279172874252 -lt 279172874252 2024-03-09T13:45:54.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:54.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089346 2024-03-09T13:45:54.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:54.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:45:54.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089346 2024-03-09T13:45:54.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:54.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089346 2024-03-09T13:45:54.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2024-03-09T13:45:54.870 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 382252089346 2024-03-09T13:45:54.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:45:55.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089346 -lt 382252089346 2024-03-09T13:45:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:45:55.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383685 2024-03-09T13:45:55.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:45:55.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:45:55.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383685 2024-03-09T13:45:55.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:45:55.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383685 2024-03-09T13:45:55.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383685' 2024-03-09T13:45:55.150 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 343597383685 2024-03-09T13:45:55.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:45:55.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383686 -lt 343597383685 2024-03-09T13:45:55.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:45:55.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:45:55.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:45:55.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:45:55.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:45:55.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:45:55.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:45:55.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:45:55.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:45:55.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:45:55.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:45:56.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:45:56.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:45:56.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:45:56.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:45:56.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:45:56.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:45:56.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:45:56.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:45:56.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-03-09T13:45:56.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-03-09T13:45:56.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:45:56.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:45:56.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-03-09T13:45:56.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:45:56.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:45:56.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:45:56.767 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:45:56.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:45:57.286 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:45:57.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:45:57.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:45:57.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:45:57.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:45:57.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:45:57.485 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:45:57.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:45:57.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:45:57.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:45:57.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:45:57.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-09T13:45:57.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:45:57.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:45:57.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:45:57.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:45:57.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:45:57.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-03-09T13:45:59.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:45:59.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:45:59.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:45:59.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:45:59.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:45:59.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:45:59.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:45:59.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:45:59.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:45:59.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:45:59.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:45:59.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:45:59.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:45:59.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:45:59.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:45:59.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:45:59.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:45:59.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:45:59.949 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:45:59.949 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:45:59.949 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:45:59.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:45:59.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:45:59.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:45:59.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:45:59.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:45:59.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:45:59.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:45:59.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:45:59.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:45:59.950 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:45:59.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:45:59.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:45:59.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:45:59.951 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:45:59.951 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:45:59.951 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:45:59.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:45:59.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:45:59.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:45:59.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:45:59.972 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:59.970+0000 7fec4d413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:59.974 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:59.972+0000 7fec4d413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:45:59.975 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:45:59.974+0000 7fec4d413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:46:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:46:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:46:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:46:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:46:00.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:46:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:46:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:46:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:00.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:46:00.491 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:00.490+0000 7fec4d413640 -1 Falling back to public interface 2024-03-09T13:46:00.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:46:01.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:46:01.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:01.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:46:01.502 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:46:01.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:01.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:46:01.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:46:02.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:46:02.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:02.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:46:02.788 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:46:02.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:02.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:46:02.930 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:02.928+0000 7fec4d413640 -1 osd.0 93 log_to_monitors true 2024-03-09T13:46:03.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:46:04.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:46:04.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:04.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:46:04.142 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:46:04.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:04.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:46:04.490 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 98 up_thru 98 down_at 94 last_clean_interval [65,93) [v2:127.0.0.1:6802/3481255862,v1:127.0.0.1:6803/3481255862] [v2:127.0.0.1:6804/3481255862,v1:127.0.0.1:6805/3481255862] exists,up 37480e6a-5ac8-4ec7-9083-5bae93ea29e1 2024-03-09T13:46:04.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:46:04.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:46:04.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:46:04.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:46:04.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:46:04.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:46:04.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:46:04.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:46:04.491 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:46:04.491 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:46:04.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:46:04.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:46:04.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:46:04.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:46:04.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:46:04.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:46:04.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:46:04.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:46:04.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:46:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:46:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:46:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:46:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:46:04.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:46:04.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:46:05.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795010 2024-03-09T13:46:05.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795010 2024-03-09T13:46:05.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010' 2024-03-09T13:46:05.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:46:05.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:46:05.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089349 2024-03-09T13:46:05.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089349 2024-03-09T13:46:05.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089349' 2024-03-09T13:46:05.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:46:05.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:46:05.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383689 2024-03-09T13:46:05.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383689 2024-03-09T13:46:05.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795010 1-382252089349 2-343597383689' 2024-03-09T13:46:05.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:46:05.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795010 2024-03-09T13:46:05.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:46:05.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:46:05.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795010 2024-03-09T13:46:05.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:46:05.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795010 2024-03-09T13:46:05.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795010' 2024-03-09T13:46:05.423 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 420906795010 2024-03-09T13:46:05.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:46:05.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 420906795010 2024-03-09T13:46:05.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:46:06.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:46:06.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:46:06.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795010 -lt 420906795010 2024-03-09T13:46:06.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:46:06.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-382252089349 2024-03-09T13:46:06.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:46:06.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:46:06.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-382252089349 2024-03-09T13:46:06.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:46:06.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089349 2024-03-09T13:46:06.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 382252089349' 2024-03-09T13:46:06.984 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 382252089349 2024-03-09T13:46:06.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:46:07.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089350 -lt 382252089349 2024-03-09T13:46:07.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:46:07.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383689 2024-03-09T13:46:07.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:46:07.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:46:07.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383689 2024-03-09T13:46:07.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:46:07.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383689 2024-03-09T13:46:07.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383689' 2024-03-09T13:46:07.270 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 343597383689 2024-03-09T13:46:07.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:46:07.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383689 -lt 343597383689 2024-03-09T13:46:07.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:46:07.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:46:07.545 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:46:07.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:46:07.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:46:07.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:46:07.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:46:07.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:46:07.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:46:07.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:46:07.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:46:08.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:46:08.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:46:08.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:46:08.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:46:08.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:46:08.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:46:08.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:46:08.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-03-09T13:46:08.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T13:46:08.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:46:08.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:46:08.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:46:08.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:46:08.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T13:46:08.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:46:08.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:46:08.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:46:08.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:46:08.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:46:08.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:46:08.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:46:08.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:46:08.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:46:08.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:46:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:46:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T13:46:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:46:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:46:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:46:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:46:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:46:08.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T13:46:10.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:46:10.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:46:10.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:46:10.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:46:10.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:46:10.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:46:10.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:46:10.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:46:10.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:46:10.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:46:10.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:46:10.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:46:10.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:46:10.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:46:10.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:46:10.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:46:10.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:46:10.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:46:10.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:46:10.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:46:10.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:46:10.903 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:46:10.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:46:10.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:46:10.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:46:10.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:46:10.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:46:10.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:46:10.932 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:10.930+0000 7fafc8a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:46:10.934 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:10.932+0000 7fafc8a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:46:10.935 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:10.934+0000 7fafc8a0c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:46:11.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:46:11.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:46:11.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:46:11.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:46:11.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:46:11.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:11.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:46:11.186 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:46:11.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:11.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:46:11.454 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:11.453+0000 7fafc8a0c640 -1 Falling back to public interface 2024-03-09T13:46:11.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:46:12.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:46:12.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:12.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:46:12.460 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:46:12.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:12.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:46:12.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:46:13.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:46:13.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:13.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:46:13.744 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:46:13.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:13.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:46:13.830 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:13.828+0000 7fafc8a0c640 -1 osd.1 101 log_to_monitors true 2024-03-09T13:46:14.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:46:15.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:46:15.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:15.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:46:15.022 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:46:15.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:15.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:46:15.414 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 106 up_thru 106 down_at 102 last_clean_interval [89,101) [v2:127.0.0.1:6810/3473926027,v1:127.0.0.1:6811/3473926027] [v2:127.0.0.1:6812/3473926027,v1:127.0.0.1:6813/3473926027] exists,up c59f0c6c-4dcd-47c5-9014-192965a74d4c 2024-03-09T13:46:15.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:46:15.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:46:15.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:46:15.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:46:15.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:46:15.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:46:15.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:46:15.415 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:46:15.416 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:46:15.416 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:46:15.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:46:15.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:46:15.416 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:46:15.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:46:15.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:46:15.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:46:15.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:46:15.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:46:15.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:46:15.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:46:15.855 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:46:15.855 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:46:15.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:46:15.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:46:15.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:46:15.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795013 2024-03-09T13:46:15.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795013 2024-03-09T13:46:15.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013' 2024-03-09T13:46:15.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:46:15.977 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:46:16.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533378 2024-03-09T13:46:16.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533378 2024-03-09T13:46:16.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013 1-455266533378' 2024-03-09T13:46:16.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:46:16.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:46:16.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383692 2024-03-09T13:46:16.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383692 2024-03-09T13:46:16.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-420906795013 1-455266533378 2-343597383692' 2024-03-09T13:46:16.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:46:16.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-420906795013 2024-03-09T13:46:16.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:46:16.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:46:16.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-420906795013 2024-03-09T13:46:16.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:46:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795013 2024-03-09T13:46:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 420906795013' 2024-03-09T13:46:16.225 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 420906795013 2024-03-09T13:46:16.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:46:16.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795012 -lt 420906795013 2024-03-09T13:46:16.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:46:17.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:46:17.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:46:17.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795013 -lt 420906795013 2024-03-09T13:46:17.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:46:17.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533378 2024-03-09T13:46:17.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:46:17.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:46:17.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533378 2024-03-09T13:46:17.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:46:17.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533378 2024-03-09T13:46:17.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533378' 2024-03-09T13:46:17.795 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 455266533378 2024-03-09T13:46:17.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:46:18.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533378 -lt 455266533378 2024-03-09T13:46:18.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:46:18.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383692 2024-03-09T13:46:18.068 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:46:18.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:46:18.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383692 2024-03-09T13:46:18.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:46:18.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383692 2024-03-09T13:46:18.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383692' 2024-03-09T13:46:18.071 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 343597383692 2024-03-09T13:46:18.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:46:18.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383692 -lt 343597383692 2024-03-09T13:46:18.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:46:18.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:46:18.346 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:46:18.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:46:18.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:46:18.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:46:18.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:46:18.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:46:18.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:46:18.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:46:18.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:46:18.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:46:18.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:46:18.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:46:18.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:46:19.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T13:46:19.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T13:46:19.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:46:19.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T13:46:19.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:46:19.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:46:19.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:46:19.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:46:19.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:46:19.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:46:19.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:46:19.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:46:19.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:46:19.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:46:19.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:46:19.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:46:19.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:46:19.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:46:20.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T13:46:20.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T13:46:20.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:46:20.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:46:20.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:46:20.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:46:20.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:46:20.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:46:20.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:46:20.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:46:20.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:46:20.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:46:20.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:46:20.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:46:20.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:46:20.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:46:20.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:46:20.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:46:20.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:46:20.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:46:20.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:46:20.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:46:20.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:46:20.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:46:20.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:46:20.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:46:21.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:46:21.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:46:21.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:46:21.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:46:21.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-03-09T13:46:21.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-03-09T13:46:21.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:46:21.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:46:21.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-03-09T13:46:21.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:46:21.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:46:21.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:46:21.541 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:46:21.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:46:22.043 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:46:22.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:46:22.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:46:22.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:46:22.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-03-09T13:46:22.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T13:46:22.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:46:22.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-09T13:46:22.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-09T13:46:22.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-03-09T13:46:22.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-09T13:46:22.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:46:22.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:46:22.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:46:22.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:46:22.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-09T13:46:22.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:46:22.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:46:22.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:46:22.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:46:22.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:46:22.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:46:22.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:46:22.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:46:22.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:46:22.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:46:22.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:46:22.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-09T13:46:22.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:46:22.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:46:22.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:46:22.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:46:22.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:46:22.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-03-09T13:46:24.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:46:24.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:46:24.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:46:24.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:46:24.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:46:24.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:46:24.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:46:24.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:46:24.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:46:24.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:46:24.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:46:24.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:46:24.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:46:24.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:46:24.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:46:24.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:46:24.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:46:24.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:46:24.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:46:24.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:46:24.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:46:24.562 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:46:24.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:46:24.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:46:24.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:46:24.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:46:24.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:46:24.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:46:24.589 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:24.588+0000 7f99424ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:46:24.591 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:24.591+0000 7f99424ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:46:24.592 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:24.592+0000 7f99424ce640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:46:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:46:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:46:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:46:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:46:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:46:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:46:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:46:24.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:24.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:46:25.107 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:25.107+0000 7f99424ce640 -1 Falling back to public interface 2024-03-09T13:46:25.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:46:26.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:46:26.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:26.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:46:26.120 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:46:26.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:26.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:46:26.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:46:27.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:46:27.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:27.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:46:27.404 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:46:27.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:27.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:46:27.429 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:27.428+0000 7f99424ce640 -1 osd.0 111 log_to_monitors true 2024-03-09T13:46:27.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:46:28.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:46:28.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:28.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:46:28.700 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:46:28.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:28.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:46:29.066 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 116 up_thru 116 down_at 112 last_clean_interval [98,111) [v2:127.0.0.1:6802/1355281935,v1:127.0.0.1:6803/1355281935] [v2:127.0.0.1:6804/1355281935,v1:127.0.0.1:6805/1355281935] exists,up 37480e6a-5ac8-4ec7-9083-5bae93ea29e1 2024-03-09T13:46:29.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:46:29.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:46:29.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:46:29.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:46:29.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:46:29.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:46:29.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:46:29.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:46:29.067 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:46:29.067 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:46:29.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:46:29.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:46:29.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:46:29.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:46:29.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:46:29.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:46:29.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:46:29.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:46:29.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:46:29.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:46:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:46:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:46:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:46:29.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:46:29.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:46:29.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=498216206338 2024-03-09T13:46:29.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 498216206338 2024-03-09T13:46:29.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338' 2024-03-09T13:46:29.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:46:29.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:46:29.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533382 2024-03-09T13:46:29.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533382 2024-03-09T13:46:29.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338 1-455266533382' 2024-03-09T13:46:29.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:46:29.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:46:29.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383696 2024-03-09T13:46:29.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383696 2024-03-09T13:46:29.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-498216206338 1-455266533382 2-343597383696' 2024-03-09T13:46:29.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:46:29.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-498216206338 2024-03-09T13:46:29.816 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:46:29.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:46:29.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-498216206338 2024-03-09T13:46:29.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:46:29.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=498216206338 2024-03-09T13:46:29.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 498216206338' 2024-03-09T13:46:29.819 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 498216206338 2024-03-09T13:46:29.819 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:46:30.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 498216206338 2024-03-09T13:46:30.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:46:31.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:46:31.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:46:31.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 498216206338 -lt 498216206338 2024-03-09T13:46:31.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:46:31.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533382 2024-03-09T13:46:31.374 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:46:31.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:46:31.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533382 2024-03-09T13:46:31.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:46:31.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533382 2024-03-09T13:46:31.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533382' 2024-03-09T13:46:31.376 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 455266533382 2024-03-09T13:46:31.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:46:31.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533382 -lt 455266533382 2024-03-09T13:46:31.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:46:31.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383696 2024-03-09T13:46:31.650 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:46:31.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:46:31.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383696 2024-03-09T13:46:31.652 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:46:31.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383696 2024-03-09T13:46:31.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383696' 2024-03-09T13:46:31.653 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 343597383696 2024-03-09T13:46:31.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:46:31.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383696 -lt 343597383696 2024-03-09T13:46:31.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:46:31.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:46:31.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:46:32.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:46:32.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:46:32.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:46:32.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:46:32.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:46:32.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:46:32.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:46:32.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:46:32.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:46:32.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:46:32.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:46:32.564 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:46:32.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:46:32.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:46:32.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:46:32.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T13:46:32.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:46:32.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:46:32.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:46:32.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:46:32.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T13:46:32.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:46:32.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:46:32.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:46:32.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:46:32.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:46:32.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:46:32.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:46:32.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:46:32.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:46:32.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:46:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:46:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T13:46:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:46:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:46:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:46:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:46:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:46:33.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T13:46:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:46:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:46:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:46:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:46:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:46:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:46:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:46:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:46:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:46:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:46:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:46:35.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:46:35.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:46:35.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:46:35.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:46:35.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:46:35.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:46:35.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:46:35.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:46:35.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:46:35.629 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:46:35.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:46:35.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:46:35.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:46:35.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:46:35.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:46:35.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:46:35.656 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:35.656+0000 7f4f7fc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:46:35.658 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:35.658+0000 7f4f7fc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:46:35.660 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:35.660+0000 7f4f7fc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:46:35.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:46:35.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:46:35.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:46:35.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:46:35.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:46:35.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:35.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:46:35.915 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:46:35.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:35.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:46:36.174 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:36.174+0000 7f4f7fc0d640 -1 Falling back to public interface 2024-03-09T13:46:36.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:46:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:46:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:46:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:46:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:46:37.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:46:38.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:46:38.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:38.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:46:38.470 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:46:38.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:38.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:46:38.475 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:38.475+0000 7f4f7fc0d640 -1 osd.0 118 log_to_monitors true 2024-03-09T13:46:38.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:46:39.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:46:39.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:46:39.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:46:39.860 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:46:39.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:46:39.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:46:40.302 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 123 up_thru 123 down_at 119 last_clean_interval [116,118) [v2:127.0.0.1:6802/2445587985,v1:127.0.0.1:6803/2445587985] [v2:127.0.0.1:6804/2445587985,v1:127.0.0.1:6805/2445587985] exists,up 37480e6a-5ac8-4ec7-9083-5bae93ea29e1 2024-03-09T13:46:40.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:46:40.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:46:40.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:46:40.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:46:40.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:46:40.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:46:40.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:46:40.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:46:40.303 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:46:40.303 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:46:40.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:46:40.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:46:40.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:46:40.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:46:40.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:46:40.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:46:40.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:46:40.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:46:40.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:46:40.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:46:40.732 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:46:40.732 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:46:40.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:46:40.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:46:40.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:46:40.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-03-09T13:46:40.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-03-09T13:46:40.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410' 2024-03-09T13:46:40.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:46:40.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:46:40.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=455266533385 2024-03-09T13:46:40.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 455266533385 2024-03-09T13:46:40.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410 1-455266533385' 2024-03-09T13:46:40.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:46:40.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:46:41.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=343597383699 2024-03-09T13:46:41.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 343597383699 2024-03-09T13:46:41.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-528280977410 1-455266533385 2-343597383699' 2024-03-09T13:46:41.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:46:41.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-528280977410 2024-03-09T13:46:41.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:46:41.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:46:41.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-528280977410 2024-03-09T13:46:41.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:46:41.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977410 2024-03-09T13:46:41.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 528280977410' 2024-03-09T13:46:41.050 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 528280977410 2024-03-09T13:46:41.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:46:41.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 528280977410 2024-03-09T13:46:41.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:46:42.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:46:42.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:46:42.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 528280977410 2024-03-09T13:46:42.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:46:43.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:46:43.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:46:43.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977410 -lt 528280977410 2024-03-09T13:46:43.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:46:43.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-455266533385 2024-03-09T13:46:43.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:46:43.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:46:43.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-455266533385 2024-03-09T13:46:43.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:46:43.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=455266533385 2024-03-09T13:46:43.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 455266533385' 2024-03-09T13:46:43.892 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 455266533385 2024-03-09T13:46:43.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:46:44.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 455266533385 -lt 455266533385 2024-03-09T13:46:44.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:46:44.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-343597383699 2024-03-09T13:46:44.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:46:44.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:46:44.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-343597383699 2024-03-09T13:46:44.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:46:44.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=343597383699 2024-03-09T13:46:44.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 343597383699' 2024-03-09T13:46:44.199 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 343597383699 2024-03-09T13:46:44.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:46:44.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 343597383699 -lt 343597383699 2024-03-09T13:46:44.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:46:44.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:46:44.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:46:44.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:46:44.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:46:44.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:46:44.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:46:44.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:46:44.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:46:44.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:46:44.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:46:45.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:46:45.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:46:45.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:46:45.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:46:45.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:46:45.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:46:45.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:46:45.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-03-09T13:46:45.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-03-09T13:46:45.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:46:45.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-03-09T13:46:45.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-03-09T13:46:45.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:46:45.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-03-09T13:46:45.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-03-09T13:46:45.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-09T13:46:45.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-09T13:46:45.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:46:45.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:46:45.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:46:45.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:46:45.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-09T13:44:23.847752+0000 2024-03-09T13:46:45.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-09T13:46:46.228 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-03-09T13:46:46.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-09T13:44:23.847752+0000 2024-03-09T13:46:46.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T13:46:46.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:44:23.847752+0000 2024-03-09T13:46:46.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:46:46.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:46:46.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:46:46.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:46:46.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:46:46.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:46:46.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:46:46.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:46:46.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:44:23.847752+0000 '>' 2024-03-09T13:44:23.847752+0000 2024-03-09T13:46:46.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:46:47.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:46:47.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:46:47.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:46:47.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:46:47.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:46:47.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:46:47.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:46:47.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:44:23.847752+0000 '>' 2024-03-09T13:44:23.847752+0000 2024-03-09T13:46:47.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:46:48.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:46:48.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:46:48.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:46:48.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:46:48.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:46:48.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:46:48.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:46:49.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:44:23.847752+0000 '>' 2024-03-09T13:44:23.847752+0000 2024-03-09T13:46:49.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:46:50.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:46:50.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:46:50.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:46:50.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:46:50.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:46:50.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:46:50.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:46:50.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:44:23.847752+0000 '>' 2024-03-09T13:44:23.847752+0000 2024-03-09T13:46:50.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:46:51.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:46:51.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:46:51.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:46:51.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:46:51.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:46:51.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:46:51.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:46:51.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:46:46.664207+0000 '>' 2024-03-09T13:44:23.847752+0000 2024-03-09T13:46:51.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:46:51.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-09T13:46:51.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-09T13:46:51.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-03-09T13:46:51.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T13:46:51.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-09T13:46:51.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-09T13:46:51.688 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-09T13:46:51.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=123 2024-03-09T13:46:51.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T13:46:51.691 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T13:46:51.691 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T13:46:51.691 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T13:46:51.691 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T13:46:51.691 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T13:46:51.691 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T13:46:51.691 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T13:46:51.691 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T13:46:51.691 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T13:46:51.691 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T13:46:51.691 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T13:46:51.692 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T13:46:51.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T13:46:51.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T13:46:51.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T13:46:51.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T13:46:51.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T13:46:51.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T13:46:51.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-03-09T13:46:51.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-03-09T13:46:51.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-03-09T13:46:51.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-09T13:46:51.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:46:51.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:46:51.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:46:51.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:46:51.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:46:52.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-09T13:44:23.847752+0000 2024-03-09T13:46:52.034 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-09T13:46:52.294 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-03-09T13:46:52.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-09T13:44:23.847752+0000 last_deep_scrub_stamp 2024-03-09T13:46:52.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T13:46:52.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:44:23.847752+0000 2024-03-09T13:46:52.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-09T13:46:52.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:46:52.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:46:52.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:46:52.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:46:52.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:46:52.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:46:52.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:46:52.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:44:23.847752+0000 '>' 2024-03-09T13:44:23.847752+0000 2024-03-09T13:46:52.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:46:53.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:46:53.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:46:53.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:46:53.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:46:53.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:46:53.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:46:53.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:46:53.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:44:23.847752+0000 '>' 2024-03-09T13:44:23.847752+0000 2024-03-09T13:46:53.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:46:54.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:46:54.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:46:54.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:46:54.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:46:54.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:46:54.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:46:54.862 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:46:55.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:46:52.703293+0000 '>' 2024-03-09T13:44:23.847752+0000 2024-03-09T13:46:55.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:46:55.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-09T13:46:55.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-09T13:46:55.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-03-09T13:46:55.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T13:46:55.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-09T13:46:55.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-09T13:46:55.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-09T13:46:55.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=123 2024-03-09T13:46:55.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' false = true ']' 2024-03-09T13:46:55.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T13:46:55.206 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T13:46:55.206 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T13:46:55.206 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T13:46:55.206 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T13:46:55.206 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T13:46:55.206 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T13:46:55.206 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T13:46:55.206 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T13:46:55.206 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T13:46:55.207 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T13:46:55.207 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T13:46:55.207 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T13:46:55.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T13:46:55.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4991: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T13:46:55.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T13:46:55.230 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T13:46:55.230 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T13:46:55.230 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T13:46:55.230 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T13:46:55.230 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T13:46:55.230 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T13:46:55.230 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T13:46:55.230 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T13:46:55.230 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T13:46:55.230 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T13:46:55.231 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T13:46:55.231 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T13:46:55.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T13:46:55.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T13:46:55.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T13:46:55.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T13:46:55.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-03-09T13:46:55.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-03-09T13:46:55.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-03-09T13:46:55.620 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' removed 2024-03-09T13:46:55.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:46:55.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:46:55.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:46:55.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:46:55.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:46:55.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:46:55.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:46:55.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:46:55.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:46:55.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:46:55.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:46:55.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:46:55.742 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:46:55.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:46:55.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:46:55.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:46:55.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:46:55.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:46:55.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:46:55.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:46:55.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:46:55.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:46:55.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:46:55.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:46:55.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:46:55.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:46:55.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:46:55.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:46:55.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:46:55.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:46:55.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:46:55.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:46:55.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:46:55.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:46:55.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:46:55.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:46:55.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:46:55.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:46:55.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:46:55.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:46:55.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:46:55.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:46:55.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:46:55.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:46:55.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:46:55.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:46:55.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:46:55.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:46:55.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:46:55.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:46:55.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:46:55.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:46:55.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:46:55.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:46:55.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:46:55.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:46:55.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:46:55.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:46:55.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:46:55.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:46:55.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:46:55.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:46:55.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:46:55.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:46:55.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:46:55.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:46:55.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:46:55.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:46:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:46:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:46:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2024-03-09T13:46:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5750: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2024-03-09T13:46:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5751: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2024-03-09T13:46:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3596: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2024-03-09T13:46:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3597: corrupt_scrub_erasure: local allow_overwrites=true 2024-03-09T13:46:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3598: corrupt_scrub_erasure: local poolname=ecpool 2024-03-09T13:46:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3599: corrupt_scrub_erasure: local total_objs=7 2024-03-09T13:46:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3601: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2024-03-09T13:46:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:46:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:46:55.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:46:55.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:46:55.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:46:55.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T13:46:56.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:46:56.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:46:56.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:46:56.253 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:46:56.253 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:46:56.253 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:46:56.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:46:56.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T13:46:56.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:46:56.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:46:56.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:46:56.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:46:56.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:46:56.456 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:46:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:46:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:46:56.456 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:46:56.457 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:46:56.457 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:46:56.457 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:46:56.457 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:46:56.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:46:56.457 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:46:56.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:46:56.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:46:56.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:46:56.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:46:56.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:46:56.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:46:56.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:46:56.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:46:56.541 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:46:56.541 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:46:56.541 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:46:56.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:46:56.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:46:56.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:46:56.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3602: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2024-03-09T13:46:56.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:46:56.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:46:56.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:46:56.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:46:56.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:46:56.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:46:57.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:46:57.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:46:57.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:46:57.431 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:46:57.431 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:46:57.431 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:46:57.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:46:57.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:46:57.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:46:57.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: seq 0 2 2024-03-09T13:46:57.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-09T13:46:57.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2024-03-09T13:46:57.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:46:57.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:46:57.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:46:57.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:46:57.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:46:57.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:46:57.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:46:57.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:46:57.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:46:57.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d7431c32-c2c6-4350-9d07-f7a8f1b761fd 2024-03-09T13:46:57.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d7431c32-c2c6-4350-9d07-f7a8f1b761fd' 2024-03-09T13:46:57.466 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 d7431c32-c2c6-4350-9d07-f7a8f1b761fd 2024-03-09T13:46:57.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:46:57.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBRaOxlubagHBAAEcgLenxE/FkRU3ic4y5amA== 2024-03-09T13:46:57.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBRaOxlubagHBAAEcgLenxE/FkRU3ic4y5amA=="}' 2024-03-09T13:46:57.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d7431c32-c2c6-4350-9d07-f7a8f1b761fd -i td/osd-scrub-repair/0/new.json 2024-03-09T13:46:57.686 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:46:57.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:46:57.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBRaOxlubagHBAAEcgLenxE/FkRU3ic4y5amA== --osd-uuid d7431c32-c2c6-4350-9d07-f7a8f1b761fd 2024-03-09T13:46:57.725 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:57.723+0000 7f6140421640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:46:57.737 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:57.736+0000 7f6140421640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:46:57.738 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:57.737+0000 7f6140421640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:46:57.739 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:57.737+0000 7f6140421640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:46:57.739 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:57.737+0000 7f6140421640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:46:57.739 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:57.737+0000 7f6140421640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:46:57.829 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:46:57.828+0000 7f6140421640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:47:03.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:47:03.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:47:03.385 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:47:03.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:47:03.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:47:03.749 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:47:03.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:47:03.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:47:03.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:47:03.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:47:03.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:47:03.775 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:03.774+0000 7f65eaaae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:47:03.777 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:03.776+0000 7f65eaaae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:47:03.778 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:03.777+0000 7f65eaaae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:47:04.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:47:04.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:47:04.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:47:04.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:47:04.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:47:04.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:04.024 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:47:04.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:47:04.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:04.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:04.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:04.301+0000 7f65eaaae640 -1 Falling back to public interface 2024-03-09T13:47:05.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:05.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:05.295 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:47:05.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:47:05.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:05.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:05.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:06.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:06.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:06.584 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:47:06.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:47:06.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:06.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:06.643 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:06.642+0000 7f65eaaae640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:47:06.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:07.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:07.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:07.912 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:47:07.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:47:07.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:07.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:08.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:09.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:09.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:09.195 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:47:09.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:47:09.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:09.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:09.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:10.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:10.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:10.584 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:47:10.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:47:10.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:10.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:10.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:11.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:11.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:11.868 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:47:11.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:47:11.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:11.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:12.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:13.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:13.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:13.241 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:47:13.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:47:13.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:13.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:13.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:14.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:14.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:14.600 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:47:14.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:47:14.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:14.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:14.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:15.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:15.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:15.882 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:47:15.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:47:15.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:15.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:16.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:17.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:17.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:17.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:47:17.248 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:47:17.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:17.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:17.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:18.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:18.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:18.582 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:47:18.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:47:18.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:18.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:19.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:20.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:20.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:20.307 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:47:20.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:47:20.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:20.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:20.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:21.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:21.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:21.775 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:47:21.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:47:21.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:21.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:22.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:23.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:23.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:23.627 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:47:23.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:47:23.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:23.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:24.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:25.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:25.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:25.346 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:47:25.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:47:25.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:25.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:25.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:26.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:26.807 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:47:26.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:26.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:47:26.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:26.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:27.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:28.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:28.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:28.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:47:28.278 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:47:28.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:28.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:28.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:29.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:29.662 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:47:29.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:29.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:47:29.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:29.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:47:29.959 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2287380620,v1:127.0.0.1:6803/2287380620] [v2:127.0.0.1:6804/2287380620,v1:127.0.0.1:6805/2287380620] exists,up d7431c32-c2c6-4350-9d07-f7a8f1b761fd 2024-03-09T13:47:29.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:47:29.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:47:29.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:47:29.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-09T13:47:29.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2024-03-09T13:47:29.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:47:29.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:47:29.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:47:29.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:47:29.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:47:29.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:47:29.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:47:29.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:47:29.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:47:29.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:47:29.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:47:29.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:47:29.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:47:29.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:47:29.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:47:29.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:47:29.968 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:47:29.968 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:47:29.968 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:47:29.969 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:47:29.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:47:29.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:47:29.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:47:29.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:47:29.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:47:29.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:47:29.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:47:29.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:47:29.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:47:29.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:47:29.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:47:29.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:47:29.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:47:29.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:47:29.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:47:29.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd73c514-d245-46ec-bdb1-70ee6e12902e 2024-03-09T13:47:29.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 dd73c514-d245-46ec-bdb1-70ee6e12902e' 2024-03-09T13:47:29.971 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 dd73c514-d245-46ec-bdb1-70ee6e12902e 2024-03-09T13:47:29.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:47:29.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxaOxlJUFzOhAA8ncLPQP6sMnLg+tefXOfPA== 2024-03-09T13:47:29.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxaOxlJUFzOhAA8ncLPQP6sMnLg+tefXOfPA=="}' 2024-03-09T13:47:29.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd73c514-d245-46ec-bdb1-70ee6e12902e -i td/osd-scrub-repair/1/new.json 2024-03-09T13:47:30.321 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:47:30.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:47:30.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxaOxlJUFzOhAA8ncLPQP6sMnLg+tefXOfPA== --osd-uuid dd73c514-d245-46ec-bdb1-70ee6e12902e 2024-03-09T13:47:30.361 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:30.360+0000 7f2fd0ad9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:47:30.405 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:30.405+0000 7f2fd0ad9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:47:30.406 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:30.406+0000 7f2fd0ad9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:47:30.407 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:30.407+0000 7f2fd0ad9640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:47:30.407 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:30.407+0000 7f2fd0ad9640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:47:30.407 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:30.407+0000 7f2fd0ad9640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:47:30.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:30.463+0000 7f2fd0ad9640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:47:36.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:47:36.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:47:36.557 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:47:36.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:47:36.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:47:36.912 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:47:36.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:47:36.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:47:36.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:47:36.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:47:36.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:47:36.938 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:36.936+0000 7f8f1fc20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:47:36.940 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:36.939+0000 7f8f1fc20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:47:36.941 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:36.940+0000 7f8f1fc20640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:47:37.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:47:37.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:47:37.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:47:37.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:47:37.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:47:37.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:37.188 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:47:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:47:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:37.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:37.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:37.471 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:37.470+0000 7f8f1fc20640 -1 Falling back to public interface 2024-03-09T13:47:38.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:38.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:38.461 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:47:38.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:47:38.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:38.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:38.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:39.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:39.745 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:47:39.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:39.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:47:39.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:39.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:39.943 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:47:39.942+0000 7f8f1fc20640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:47:40.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:41.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:41.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:41.044 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:47:41.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:47:41.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:41.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:41.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:42.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:42.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:42.327 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:47:42.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:47:42.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:42.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:42.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:43.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:43.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:43.609 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:47:43.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:47:43.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:43.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:43.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:44.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:44.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:44.898 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:47:44.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:47:44.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:44.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:45.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:46.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:46.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:46.183 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:47:46.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:47:46.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:46.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:46.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:47.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:47.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:47.472 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:47:47.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:47:47.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:47.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:47.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:48.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:48.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:48.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:47:48.854 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:47:48.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:48.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:49.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:50.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:50.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:50.139 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:47:50.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:47:50.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:50.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:50.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:51.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:51.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:51.428 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:47:51.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:47:51.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:51.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:51.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:52.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:52.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:53.000 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:47:53.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:47:53.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:53.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:53.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:54.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:54.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:54.615 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:47:54.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:47:54.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:54.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:55.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:56.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:56.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:56.144 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:47:56.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:47:56.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:56.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:56.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:57.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:57.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:57.475 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:47:57.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:47:57.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:57.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:57.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:47:58.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:47:58.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:47:58.956 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:47:58.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:47:58.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:47:58.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:47:59.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:00.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:00.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:00.568 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:48:00.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:48:00.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:00.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:48:01.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:02.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:02.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:02.143 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:48:02.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:48:02.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:02.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:48:02.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:03.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:03.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:03.495 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:48:03.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:48:03.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:03.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:48:03.777 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1029781713,v1:127.0.0.1:6811/1029781713] [v2:127.0.0.1:6812/1029781713,v1:127.0.0.1:6813/1029781713] exists,up dd73c514-d245-46ec-bdb1-70ee6e12902e 2024-03-09T13:48:03.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:48:03.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:48:03.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:48:03.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3603: corrupt_scrub_erasure: for id in $(seq 0 2) 2024-03-09T13:48:03.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3604: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2024-03-09T13:48:03.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:48:03.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:48:03.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T13:48:03.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:48:03.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:48:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:48:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:48:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:48:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:48:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:48:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:48:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:48:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:48:03.783 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:48:03.783 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:48:03.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:48:03.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:48:03.784 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:48:03.784 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:48:03.784 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:48:03.784 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:48:03.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:48:03.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:48:03.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:48:03.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:48:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:48:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:48:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:48:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:48:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:48:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:48:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:48:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:48:03.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:48:03.785 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:48:03.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4da9dd03-8db5-4294-9665-3345b5cecedf 2024-03-09T13:48:03.786 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 4da9dd03-8db5-4294-9665-3345b5cecedf 2024-03-09T13:48:03.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4da9dd03-8db5-4294-9665-3345b5cecedf' 2024-03-09T13:48:03.786 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:48:03.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTaOxllyCTLxAAlOlfugRX3JI/yxrREv8+Qg== 2024-03-09T13:48:03.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTaOxllyCTLxAAlOlfugRX3JI/yxrREv8+Qg=="}' 2024-03-09T13:48:03.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4da9dd03-8db5-4294-9665-3345b5cecedf -i td/osd-scrub-repair/2/new.json 2024-03-09T13:48:04.161 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:48:04.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T13:48:04.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTaOxllyCTLxAAlOlfugRX3JI/yxrREv8+Qg== --osd-uuid 4da9dd03-8db5-4294-9665-3345b5cecedf 2024-03-09T13:48:04.201 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:04.201+0000 7f0112cac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:48:04.246 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:04.246+0000 7f0112cac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:48:04.247 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:04.247+0000 7f0112cac640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:48:04.247 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:04.247+0000 7f0112cac640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:48:04.247 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:04.247+0000 7f0112cac640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:48:04.248 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:04.247+0000 7f0112cac640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T13:48:04.321 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:04.320+0000 7f0112cac640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T13:48:10.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T13:48:10.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:48:10.442 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T13:48:10.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T13:48:10.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:48:10.801 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T13:48:10.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T13:48:10.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:48:10.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:48:10.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:48:10.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:48:10.827 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:10.827+0000 7fca4b69b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:48:10.829 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:10.829+0000 7fca4b69b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:48:10.830 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:10.830+0000 7fca4b69b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:48:11.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T13:48:11.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:48:11.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:48:11.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:48:11.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:48:11.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:11.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:48:11.077 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:48:11.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:11.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:11.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:11.350 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:11.350+0000 7fca4b69b640 -1 Falling back to public interface 2024-03-09T13:48:12.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:12.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:12.349 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:48:12.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:48:12.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:12.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:12.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:13.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:13.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:13.629 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:48:13.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:48:13.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:13.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:13.882 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:13.881+0000 7fca4b69b640 -1 osd.2 0 log_to_monitors true 2024-03-09T13:48:13.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:14.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:14.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:14.982 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:48:14.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:48:14.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:14.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:15.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:16.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:16.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:16.271 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:48:16.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:48:16.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:16.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:16.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:17.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:17.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:17.680 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:48:17.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:48:17.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:17.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:18.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:19.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:19.052 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:48:19.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:19.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:48:19.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:19.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:19.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:20.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:20.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:20.335 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:48:20.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:48:20.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:20.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:21.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:21.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:21.730 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:48:21.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:48:21.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:21.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:22.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:23.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:23.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:23.103 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:48:23.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:48:23.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:23.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:23.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:24.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:24.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:24.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:48:24.392 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:48:24.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:24.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:24.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:25.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:25.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:25.700 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:48:25.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:48:25.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:25.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:26.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:27.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:27.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:27.402 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:48:27.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:48:27.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:27.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:28.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:29.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:29.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:29.188 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:48:29.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:48:29.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:29.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:29.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:30.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:30.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:30.754 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:48:30.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:48:30.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:30.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:31.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:32.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:32.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:32.532 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:48:32.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:48:32.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:32.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:32.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:33.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:33.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:33.869 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:48:33.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:48:33.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:33.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:34.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:35.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:35.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:35.581 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:48:35.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:48:35.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:35.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:36.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:37.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:37.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:37.336 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:48:37.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:48:37.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:37.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:48:37.649 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3211258005,v1:127.0.0.1:6819/3211258005] [v2:127.0.0.1:6820/3211258005,v1:127.0.0.1:6821/3211258005] exists,up 4da9dd03-8db5-4294-9665-3345b5cecedf 2024-03-09T13:48:37.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:48:37.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:48:37.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:48:37.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3606: corrupt_scrub_erasure: create_rbd_pool 2024-03-09T13:48:37.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T13:48:37.934 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T13:48:37.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T13:48:37.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T13:48:38.347 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T13:48:38.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:48:39.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T13:48:41.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3607: corrupt_scrub_erasure: create_pool foo 1 2024-03-09T13:48:41.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-09T13:48:41.568 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'foo' created 2024-03-09T13:48:41.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:48:42.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3609: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2024-03-09T13:48:42.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T13:48:42.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T13:48:42.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-09T13:48:42.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T13:48:42.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2024-03-09T13:48:42.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T13:48:42.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T13:48:43.981 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' created 2024-03-09T13:48:43.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:48:44.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-09T13:48:44.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-09T13:48:45.498 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 3 allow_ec_overwrites to true 2024-03-09T13:48:45.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T13:48:45.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:48:45.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:48:45.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:48:45.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:48:45.511 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:48:45.511 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:48:45.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:48:45.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:48:45.512 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:48:45.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:48:45.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:48:45.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:48:45.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:48:45.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:48:45.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:48:45.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:48:45.905 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:48:45.905 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:48:45.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:48:45.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:48:45.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:48:46.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836501 2024-03-09T13:48:46.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836501 2024-03-09T13:48:46.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501' 2024-03-09T13:48:46.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:48:46.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:48:46.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672973 2024-03-09T13:48:46.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672973 2024-03-09T13:48:46.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672973' 2024-03-09T13:48:46.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:48:46.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:48:46.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509449 2024-03-09T13:48:46.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509449 2024-03-09T13:48:46.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836501 1-42949672973 2-64424509449' 2024-03-09T13:48:46.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:48:46.218 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836501 2024-03-09T13:48:46.218 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:48:46.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:48:46.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836501 2024-03-09T13:48:46.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:48:46.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836501 2024-03-09T13:48:46.221 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836501 2024-03-09T13:48:46.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-03-09T13:48:46.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:48:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836501 2024-03-09T13:48:46.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:48:47.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:48:47.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:48:47.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836501 2024-03-09T13:48:47.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:48:47.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672973 2024-03-09T13:48:47.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:48:47.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:48:47.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672973 2024-03-09T13:48:47.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:48:47.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672973 2024-03-09T13:48:47.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-03-09T13:48:47.858 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672973 2024-03-09T13:48:47.858 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:48:48.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672973 -lt 42949672973 2024-03-09T13:48:48.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:48:48.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509449 2024-03-09T13:48:48.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:48:48.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:48:48.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509449 2024-03-09T13:48:48.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:48:48.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509449 2024-03-09T13:48:48.139 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-03-09T13:48:48.139 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509449 2024-03-09T13:48:48.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:48:48.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509450 -lt 64424509449 2024-03-09T13:48:48.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:48:48.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:48:48.414 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:48:48.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:48:48.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:48:48.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:48:48.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:48:48.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:48:48.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:48:48.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:48:48.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:48:49.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:48:49.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:48:49.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:48:49.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:48:49.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:48:49.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:48:49.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:48:49.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T13:48:49.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3610: corrupt_scrub_erasure: wait_for_clean 2024-03-09T13:48:49.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:48:49.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:48:49.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:48:49.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:48:49.366 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:48:49.366 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:48:49.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:48:49.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:48:49.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:48:49.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:48:49.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:48:49.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:48:49.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:48:49.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:48:49.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:48:49.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:48:49.724 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:48:49.724 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:48:49.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:48:49.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:48:49.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:48:49.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836503 2024-03-09T13:48:49.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836503 2024-03-09T13:48:49.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503' 2024-03-09T13:48:49.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:48:49.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:48:49.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-09T13:48:49.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-09T13:48:49.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672975' 2024-03-09T13:48:49.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:48:49.952 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:48:50.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509451 2024-03-09T13:48:50.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509451 2024-03-09T13:48:50.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836503 1-42949672975 2-64424509451' 2024-03-09T13:48:50.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:48:50.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836503 2024-03-09T13:48:50.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:48:50.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:48:50.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836503 2024-03-09T13:48:50.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:48:50.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836503 2024-03-09T13:48:50.058 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836503 2024-03-09T13:48:50.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-03-09T13:48:50.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:48:50.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836503 2024-03-09T13:48:50.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:48:51.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:48:51.334 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:48:51.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836503 2024-03-09T13:48:51.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:48:51.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-09T13:48:51.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:48:51.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:48:51.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-09T13:48:51.644 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:48:51.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-09T13:48:51.645 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672975 2024-03-09T13:48:51.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-09T13:48:51.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:48:51.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-09T13:48:51.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:48:51.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509451 2024-03-09T13:48:51.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:48:51.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:48:51.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509451 2024-03-09T13:48:51.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:48:51.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509451 2024-03-09T13:48:51.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-03-09T13:48:51.924 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509451 2024-03-09T13:48:51.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:48:52.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509451 -lt 64424509451 2024-03-09T13:48:52.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:48:52.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:48:52.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:48:52.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:48:52.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:48:52.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:48:52.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:48:52.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:48:52.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:48:52.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:48:52.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:48:52.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:48:52.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:48:52.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:48:52.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:48:53.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:48:53.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:48:53.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:48:53.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: seq 1 7 2024-03-09T13:48:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:48:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ1 2024-03-09T13:48:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2024-03-09T13:48:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:48:53.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:48:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2024-03-09T13:48:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:48:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:48:53.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:48:53.595 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:48:53.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:48:54.151 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:48:54.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:48:54.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:48:54.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:48:54.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 1 % 2 2024-03-09T13:48:54.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T13:48:54.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:48:54.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3621: corrupt_scrub_erasure: local payload=UVWXYZZZ 2024-03-09T13:48:54.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3622: corrupt_scrub_erasure: echo UVWXYZZZ 2024-03-09T13:48:54.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3623: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:48:54.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:48:54.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:48:54.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:48:54.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:48:54.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:48:54.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:48:54.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:48:54.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:48:54.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:48:54.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:48:54.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:48:54.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:48:54.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:48:54.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:48:54.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:48:54.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:48:54.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:48:54.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:48:54.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:48:54.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:48:54.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:48:54.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:48:54.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:48:57.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:48:57.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:48:57.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:48:57.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:48:57.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:48:57.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:48:57.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:48:57.122 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:48:57.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:48:57.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:48:57.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:48:57.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:48:57.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:48:57.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:48:57.150 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:57.149+0000 7f2e5c615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:48:57.152 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:57.152+0000 7f2e5c615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:48:57.153 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:57.153+0000 7f2e5c615640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:48:57.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:48:57.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:48:57.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:48:57.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:48:57.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:48:57.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:57.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:48:57.408 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:48:57.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:57.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:48:57.671 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:48:57.671+0000 7f2e5c615640 -1 Falling back to public interface 2024-03-09T13:48:57.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:58.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:58.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:58.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:48:58.709 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:48:58.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:58.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:48:58.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:48:59.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:48:59.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:48:59.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:48:59.997 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:48:59.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:48:59.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:49:00.108 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:00.107+0000 7f2e5c615640 -1 osd.1 35 log_to_monitors true 2024-03-09T13:49:00.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:49:01.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:49:01.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:01.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:49:01.313 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:49:01.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:01.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:49:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/1378854136,v1:127.0.0.1:6811/1378854136] [v2:127.0.0.1:6812/1378854136,v1:127.0.0.1:6813/1378854136] exists,up dd73c514-d245-46ec-bdb1-70ee6e12902e 2024-03-09T13:49:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:49:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:49:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:49:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:49:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:49:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:49:01.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:49:01.604 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:49:01.605 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:49:01.605 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:49:01.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:49:01.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:49:01.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:49:01.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:49:01.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:49:01.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:49:01.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:49:01.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:49:01.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:49:02.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:49:02.124 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:49:02.124 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:49:02.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:49:02.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:02.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:49:02.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836506 2024-03-09T13:49:02.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836506 2024-03-09T13:49:02.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506' 2024-03-09T13:49:02.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:02.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:49:02.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691842 2024-03-09T13:49:02.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691842 2024-03-09T13:49:02.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-171798691842' 2024-03-09T13:49:02.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:02.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:49:02.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509455 2024-03-09T13:49:02.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509455 2024-03-09T13:49:02.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836506 1-171798691842 2-64424509455' 2024-03-09T13:49:02.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:02.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836506 2024-03-09T13:49:02.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:02.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:49:02.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836506 2024-03-09T13:49:02.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:02.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836506 2024-03-09T13:49:02.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-03-09T13:49:02.440 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836506 2024-03-09T13:49:02.440 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:49:02.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836505 -lt 21474836506 2024-03-09T13:49:02.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:49:03.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:49:03.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:49:04.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836506 2024-03-09T13:49:04.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:04.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691842 2024-03-09T13:49:04.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:04.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:49:04.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691842 2024-03-09T13:49:04.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:04.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691842 2024-03-09T13:49:04.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2024-03-09T13:49:04.004 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 171798691842 2024-03-09T13:49:04.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:49:04.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691842 -lt 171798691842 2024-03-09T13:49:04.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:04.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509455 2024-03-09T13:49:04.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:04.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:49:04.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509455 2024-03-09T13:49:04.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:04.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509455 2024-03-09T13:49:04.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-03-09T13:49:04.282 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509455 2024-03-09T13:49:04.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:49:04.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509455 -lt 64424509455 2024-03-09T13:49:04.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:49:04.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:49:04.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:49:04.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:49:04.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:49:04.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:49:04.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:49:04.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:49:04.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:49:04.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:49:04.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:49:05.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:49:05.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:49:05.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:49:05.173 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:49:05.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:49:05.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:49:05.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:49:05.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:49:05.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ2 2024-03-09T13:49:05.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2024-03-09T13:49:05.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:49:05.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:49:05.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2024-03-09T13:49:05.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:49:05.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:49:05.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:49:05.892 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:49:05.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:49:06.395 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:49:06.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:49:06.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:49:06.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:49:06.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 2 % 2 2024-03-09T13:49:06.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-09T13:49:06.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:49:06.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3628: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2024-03-09T13:49:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records in 2024-03-09T13:49:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:1+0 records out 2024-03-09T13:49:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 6.5317e-05 s, 31.4 MB/s 2024-03-09T13:49:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3629: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:49:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:49:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:49:06.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:49:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:49:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:49:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:49:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:49:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:49:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:49:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:49:06.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:49:06.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:49:06.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:49:06.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:49:06.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:49:06.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:49:06.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:49:06.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:49:06.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:49:06.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:49:06.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:49:06.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:49:06.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:49:09.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:49:09.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:49:09.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:49:09.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:49:09.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:49:09.178 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:49:09.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:49:09.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:49:09.181 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:49:09.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:49:09.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:49:09.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:49:09.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:49:09.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:49:09.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:49:09.208 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:09.206+0000 7f52c8a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:09.210 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:09.208+0000 7f52c8a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:09.211 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:09.210+0000 7f52c8a11640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:09.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:49:09.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:49:09.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:49:09.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:49:09.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:49:09.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:09.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:49:09.463 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:49:09.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:09.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:49:09.727 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:09.726+0000 7f52c8a11640 -1 Falling back to public interface 2024-03-09T13:49:09.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:49:10.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:49:10.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:10.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:49:10.743 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:49:10.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:10.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:49:11.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:49:12.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:49:12.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:12.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:49:12.027 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:49:12.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:12.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:49:12.144 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:12.143+0000 7f52c8a11640 -1 osd.0 44 log_to_monitors true 2024-03-09T13:49:12.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:49:13.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:49:13.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:13.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:49:13.355 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:49:13.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:13.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:49:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 49 up_thru 49 down_at 45 last_clean_interval [5,44) [v2:127.0.0.1:6802/2870260311,v1:127.0.0.1:6803/2870260311] [v2:127.0.0.1:6804/2870260311,v1:127.0.0.1:6805/2870260311] exists,up d7431c32-c2c6-4350-9d07-f7a8f1b761fd 2024-03-09T13:49:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:49:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:49:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:49:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:49:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:49:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:49:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:49:13.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:49:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:49:13.713 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:49:13.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:49:13.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:49:13.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:49:13.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:49:13.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:49:13.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:49:13.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:49:13.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:49:13.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:49:14.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:49:14.161 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:49:14.161 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:49:14.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:49:14.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:14.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:49:14.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397506 2024-03-09T13:49:14.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397506 2024-03-09T13:49:14.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506' 2024-03-09T13:49:14.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:14.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:49:14.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691845 2024-03-09T13:49:14.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691845 2024-03-09T13:49:14.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506 1-171798691845' 2024-03-09T13:49:14.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:14.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:49:14.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-03-09T13:49:14.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-03-09T13:49:14.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397506 1-171798691845 2-64424509458' 2024-03-09T13:49:14.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:14.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-210453397506 2024-03-09T13:49:14.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:14.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:49:14.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-210453397506 2024-03-09T13:49:14.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:14.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397506 2024-03-09T13:49:14.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 210453397506' 2024-03-09T13:49:14.492 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 210453397506 2024-03-09T13:49:14.492 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:49:14.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 210453397506 2024-03-09T13:49:14.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:49:15.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:49:15.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:49:16.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397506 -lt 210453397506 2024-03-09T13:49:16.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:16.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-171798691845 2024-03-09T13:49:16.043 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:16.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:49:16.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-171798691845 2024-03-09T13:49:16.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:16.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691845 2024-03-09T13:49:16.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 171798691845' 2024-03-09T13:49:16.046 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 171798691845 2024-03-09T13:49:16.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:49:16.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691846 -lt 171798691845 2024-03-09T13:49:16.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:16.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-03-09T13:49:16.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:16.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:49:16.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-03-09T13:49:16.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:16.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-03-09T13:49:16.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-03-09T13:49:16.323 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509458 2024-03-09T13:49:16.324 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:49:16.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-03-09T13:49:16.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:49:16.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:49:16.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:49:16.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:49:16.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:49:16.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:49:16.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:49:16.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:49:16.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:49:16.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:49:16.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:49:17.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:49:17.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:49:17.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:49:17.208 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:49:17.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:49:17.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:49:17.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:49:17.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:49:17.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ3 2024-03-09T13:49:17.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2024-03-09T13:49:17.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:49:17.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:49:17.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2024-03-09T13:49:17.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:49:17.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:49:17.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:49:17.933 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:49:17.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:49:18.494 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:49:18.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:49:18.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:49:18.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:49:18.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 3 % 2 2024-03-09T13:49:18.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T13:49:18.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:49:18.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3634: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2024-03-09T13:49:18.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:49:18.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:49:18.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:49:18.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:49:18.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2024-03-09T13:49:18.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:49:18.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:49:18.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:49:18.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:49:18.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:49:18.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:49:18.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:49:18.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:49:18.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:49:18.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:49:19.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:49:19.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2024-03-09T13:49:19.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:49:19.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:49:19.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:49:19.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:49:19.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:49:19.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2024-03-09T13:49:20.699 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2024-03-09T13:49:21.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:49:21.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:49:21.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:49:21.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:49:21.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:49:21.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:49:21.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:49:21.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:49:21.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:49:21.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:49:21.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:49:21.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:49:21.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:49:21.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:49:21.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:49:21.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:49:21.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:49:21.222 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:49:21.223 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:49:21.223 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:49:21.223 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:49:21.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:49:21.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:49:21.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:49:21.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:49:21.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:49:21.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:49:21.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:49:21.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:49:21.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:49:21.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:49:21.224 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:49:21.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:49:21.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:49:21.225 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:49:21.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:49:21.225 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:49:21.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:49:21.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:49:21.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:49:21.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:49:21.246 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:21.245+0000 7f0322011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:21.248 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:21.247+0000 7f0322011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:21.250 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:21.248+0000 7f0322011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:21.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:49:21.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:49:21.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:49:21.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:49:21.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:49:21.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:21.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:49:21.512 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:49:21.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:21.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:49:21.771 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:21.770+0000 7f0322011640 -1 Falling back to public interface 2024-03-09T13:49:21.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:49:22.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:49:22.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:22.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:49:22.787 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:49:22.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:22.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:49:23.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:49:24.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:49:24.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:24.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:49:24.072 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:49:24.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:24.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:49:24.091 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:24.090+0000 7f0322011640 -1 osd.1 53 log_to_monitors true 2024-03-09T13:49:24.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:49:25.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:49:25.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:25.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:49:25.527 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:49:25.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:25.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:49:26.045 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6810/3498535817,v1:127.0.0.1:6811/3498535817] [v2:127.0.0.1:6812/3498535817,v1:127.0.0.1:6813/3498535817] exists,up dd73c514-d245-46ec-bdb1-70ee6e12902e 2024-03-09T13:49:26.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:49:26.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:49:26.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:49:26.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:49:26.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:49:26.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:49:26.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:49:26.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:49:26.046 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:49:26.046 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:49:26.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:49:26.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:49:26.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:49:26.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:49:26.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:49:26.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:49:26.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:49:26.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:49:26.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:49:26.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:49:26.404 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:49:26.405 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:49:26.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:49:26.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:26.405 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:49:26.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397509 2024-03-09T13:49:26.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397509 2024-03-09T13:49:26.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397509' 2024-03-09T13:49:26.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:26.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:49:26.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103170 2024-03-09T13:49:26.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103170 2024-03-09T13:49:26.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397509 1-249108103170' 2024-03-09T13:49:26.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:26.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:49:26.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-03-09T13:49:26.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-03-09T13:49:26.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-210453397509 1-249108103170 2-64424509461' 2024-03-09T13:49:26.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:26.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-210453397509 2024-03-09T13:49:26.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:26.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:49:26.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-210453397509 2024-03-09T13:49:26.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:26.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397509 2024-03-09T13:49:26.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 210453397509' 2024-03-09T13:49:26.720 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 210453397509 2024-03-09T13:49:26.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:49:26.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397508 -lt 210453397509 2024-03-09T13:49:26.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:49:27.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:49:27.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:49:28.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397510 -lt 210453397509 2024-03-09T13:49:28.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:28.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103170 2024-03-09T13:49:28.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:28.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:49:28.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103170 2024-03-09T13:49:28.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:28.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103170 2024-03-09T13:49:28.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103170' 2024-03-09T13:49:28.277 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 249108103170 2024-03-09T13:49:28.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:49:28.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103170 -lt 249108103170 2024-03-09T13:49:28.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:28.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-03-09T13:49:28.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:28.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:49:28.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-03-09T13:49:28.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:28.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-03-09T13:49:28.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-03-09T13:49:28.556 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509461 2024-03-09T13:49:28.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:49:28.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509461 2024-03-09T13:49:28.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:49:28.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:49:28.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:49:29.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:49:29.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:49:29.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:49:29.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:49:29.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:49:29.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:49:29.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:49:29.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:49:29.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:49:29.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:49:29.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:49:29.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:49:29.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:49:29.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:49:29.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:49:29.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:49:29.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ4 2024-03-09T13:49:29.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2024-03-09T13:49:29.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:49:29.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:49:29.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2024-03-09T13:49:29.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:49:29.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:49:29.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:49:30.222 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:49:30.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:49:30.749 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:49:30.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:49:30.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:49:30.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:49:31.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 4 % 2 2024-03-09T13:49:31.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-09T13:49:31.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:49:31.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3638: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2024-03-09T13:49:31.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3639: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2024-03-09T13:49:31.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3642: corrupt_scrub_erasure: echo -n bad-val 2024-03-09T13:49:31.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3643: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T13:49:31.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:49:31.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:49:31.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:49:31.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:49:31.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T13:49:31.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:49:31.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:49:31.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:49:31.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:49:31.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:49:31.375 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:49:31.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:49:31.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:49:31.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:49:31.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:49:31.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:49:31.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T13:49:31.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:49:31.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:49:31.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:49:31.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:49:31.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:49:31.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2024-03-09T13:49:33.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:49:33.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:49:33.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:49:33.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:49:33.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:49:33.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:49:33.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:49:33.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:49:33.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:49:33.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:49:33.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:49:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:49:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:49:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:49:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:49:33.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:49:33.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:49:33.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:49:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:49:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:49:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:49:33.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:49:33.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:49:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:49:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:49:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:49:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:49:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:49:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:49:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:49:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:49:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:49:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:49:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:49:33.611 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:49:33.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:49:33.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:49:33.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:49:33.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:49:33.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:49:33.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:49:33.638 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:33.638+0000 7f09e59aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:33.642 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:33.642+0000 7f09e59aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:33.644 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:33.643+0000 7f09e59aa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:33.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:49:33.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:49:33.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:49:33.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:49:33.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:49:33.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:33.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:49:33.895 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:49:33.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:33.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:49:34.159 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:34.159+0000 7f09e59aa640 -1 Falling back to public interface 2024-03-09T13:49:34.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:49:35.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:49:35.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:35.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:49:35.167 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:49:35.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:35.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:49:35.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:49:36.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:49:36.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:36.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:49:36.450 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:49:36.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:36.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:49:36.474 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:36.472+0000 7f09e59aa640 -1 osd.0 62 log_to_monitors true 2024-03-09T13:49:36.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:49:37.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:49:37.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:37.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:49:37.844 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:49:37.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:37.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:49:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 66 up_thru 66 down_at 63 last_clean_interval [49,62) [v2:127.0.0.1:6802/1766620750,v1:127.0.0.1:6803/1766620750] [v2:127.0.0.1:6804/1766620750,v1:127.0.0.1:6805/1766620750] exists,up d7431c32-c2c6-4350-9d07-f7a8f1b761fd 2024-03-09T13:49:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:49:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:49:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:49:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:49:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:49:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:49:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:49:38.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:49:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:49:38.210 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:49:38.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:49:38.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:49:38.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:49:38.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:49:38.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:49:38.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:49:38.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:49:38.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:49:38.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:49:38.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:49:38.615 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:49:38.616 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:49:38.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:49:38.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:38.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:49:38.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-03-09T13:49:38.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-03-09T13:49:38.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538' 2024-03-09T13:49:38.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:38.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:49:38.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103173 2024-03-09T13:49:38.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103173 2024-03-09T13:49:38.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-249108103173' 2024-03-09T13:49:38.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:38.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:49:38.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509465 2024-03-09T13:49:38.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509465 2024-03-09T13:49:38.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841538 1-249108103173 2-64424509465' 2024-03-09T13:49:38.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:38.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841538 2024-03-09T13:49:38.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:38.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:49:38.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841538 2024-03-09T13:49:38.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:38.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-03-09T13:49:38.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841538' 2024-03-09T13:49:38.930 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 283467841538 2024-03-09T13:49:38.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:49:39.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 283467841538 2024-03-09T13:49:39.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:49:40.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:49:40.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:49:40.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841538 -lt 283467841538 2024-03-09T13:49:40.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:40.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103173 2024-03-09T13:49:40.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:40.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:49:40.492 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103173 2024-03-09T13:49:40.492 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:40.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103173 2024-03-09T13:49:40.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103173' 2024-03-09T13:49:40.493 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 249108103173 2024-03-09T13:49:40.493 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:49:40.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103174 -lt 249108103173 2024-03-09T13:49:40.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:40.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509465 2024-03-09T13:49:40.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:40.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:49:40.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509465 2024-03-09T13:49:40.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:40.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509465 2024-03-09T13:49:40.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-03-09T13:49:40.771 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 64424509465 2024-03-09T13:49:40.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:49:41.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509465 -lt 64424509465 2024-03-09T13:49:41.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:49:41.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:49:41.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:49:41.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:49:41.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:49:41.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:49:41.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:49:41.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:49:41.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:49:41.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:49:41.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:49:41.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:49:41.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:49:41.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:49:41.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:49:41.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T13:49:42.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-03-09T13:49:42.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:49:42.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-03-09T13:49:42.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:49:42.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:49:42.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:49:42.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:49:42.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:49:42.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:49:42.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:49:42.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:49:42.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:49:42.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:49:42.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:49:42.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:49:42.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:49:42.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:49:42.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 6 2024-03-09T13:49:42.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-03-09T13:49:42.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:49:42.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:49:42.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:49:42.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:49:42.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:49:42.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:49:42.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:49:43.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:49:43.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:49:43.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:49:43.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:49:43.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:49:43.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:49:43.252 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:49:43.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:49:43.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:49:43.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:49:43.252 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:49:43.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:49:43.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:49:43.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:49:43.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:49:43.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:49:43.521 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:49:43.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:49:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:49:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:49:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3644: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T13:49:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:49:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:49:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T13:49:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:49:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T13:49:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:49:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:49:43.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T13:49:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:49:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T13:49:43.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:49:43.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:49:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:49:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:49:43.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:49:44.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:49:44.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T13:49:44.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:49:44.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:49:44.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T13:49:44.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:49:44.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:49:44.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2024-03-09T13:49:46.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T13:49:46.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:49:46.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:49:46.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:49:46.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:49:46.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:49:46.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T13:49:46.399 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-03-09T13:49:46.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:49:46.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T13:49:46.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T13:49:46.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:49:46.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:49:46.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:49:46.426 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:46.426+0000 7f6f082ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:46.429 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:46.429+0000 7f6f082ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:46.430 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:46.430+0000 7f6f082ae640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:46.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T13:49:46.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:49:46.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:49:46.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:49:46.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:49:46.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:46.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:49:46.684 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:49:46.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:46.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:49:46.948 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:46.948+0000 7f6f082ae640 -1 Falling back to public interface 2024-03-09T13:49:46.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:49:47.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:49:47.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:47.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:49:47.957 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:49:47.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:47.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:49:48.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:49:49.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:49:49.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:49.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:49:49.236 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:49:49.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:49.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:49:49.308 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:49.307+0000 7f6f082ae640 -1 osd.2 69 log_to_monitors true 2024-03-09T13:49:49.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:49:50.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:49:50.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:50.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:49:50.541 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:49:50.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:50.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:49:50.985 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 74 up_thru 74 down_at 70 last_clean_interval [15,69) [v2:127.0.0.1:6818/827747615,v1:127.0.0.1:6819/827747615] [v2:127.0.0.1:6820/827747615,v1:127.0.0.1:6821/827747615] exists,up 4da9dd03-8db5-4294-9665-3345b5cecedf 2024-03-09T13:49:50.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:49:50.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:49:50.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:49:50.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:49:50.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:49:50.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:49:50.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:49:50.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:49:50.986 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:49:50.986 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:49:50.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:49:50.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:49:50.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:49:51.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:49:51.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:49:51.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:49:51.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:49:51.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:49:51.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:49:51.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:49:51.453 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:49:51.453 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:49:51.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:49:51.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:51.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:49:51.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841541 2024-03-09T13:49:51.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841541 2024-03-09T13:49:51.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541' 2024-03-09T13:49:51.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:51.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:49:51.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103177 2024-03-09T13:49:51.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103177 2024-03-09T13:49:51.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-249108103177' 2024-03-09T13:49:51.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:49:51.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:49:51.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=317827579906 2024-03-09T13:49:51.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 317827579906 2024-03-09T13:49:51.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841541 1-249108103177 2-317827579906' 2024-03-09T13:49:51.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:51.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841541 2024-03-09T13:49:51.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:51.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:49:51.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841541 2024-03-09T13:49:51.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:51.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841541 2024-03-09T13:49:51.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841541' 2024-03-09T13:49:51.779 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 283467841541 2024-03-09T13:49:51.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:49:52.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841540 -lt 283467841541 2024-03-09T13:49:52.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:49:53.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:49:53.052 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:49:53.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841540 -lt 283467841541 2024-03-09T13:49:53.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:49:54.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:49:54.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:49:54.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841542 -lt 283467841541 2024-03-09T13:49:54.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:54.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103177 2024-03-09T13:49:54.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:54.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:49:54.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103177 2024-03-09T13:49:54.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:54.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103177 2024-03-09T13:49:54.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103177' 2024-03-09T13:49:54.624 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 249108103177 2024-03-09T13:49:54.625 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:49:54.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103177 -lt 249108103177 2024-03-09T13:49:54.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:49:54.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-317827579906 2024-03-09T13:49:54.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:49:54.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:49:54.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-317827579906 2024-03-09T13:49:54.900 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:49:54.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=317827579906 2024-03-09T13:49:54.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 317827579906' 2024-03-09T13:49:54.901 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 317827579906 2024-03-09T13:49:54.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:49:55.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 317827579906 -lt 317827579906 2024-03-09T13:49:55.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:49:55.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:49:55.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:49:55.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:49:55.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:49:55.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:49:55.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:49:55.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:49:55.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:49:55.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:49:55.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:49:55.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:49:55.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:49:55.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:49:55.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:49:56.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:49:56.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:49:56.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:49:56.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3645: corrupt_scrub_erasure: echo -n val3-EOBJ4 2024-03-09T13:49:56.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3646: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T13:49:56.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:49:56.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:49:56.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-03-09T13:49:56.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:49:56.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T13:49:56.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:49:56.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:49:56.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-03-09T13:49:56.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:49:56.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T13:49:56.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:49:56.129 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:49:56.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:49:56.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:49:56.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:49:56.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:49:56.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T13:49:56.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:49:56.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:49:56.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-03-09T13:49:56.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:49:56.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:49:56.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2024-03-09T13:49:58.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2024-03-09T13:49:58.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:49:58.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:49:58.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T13:49:58.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:49:58.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T13:49:58.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:49:58.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:49:58.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:49:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:49:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:49:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:49:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:49:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:49:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:49:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:49:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:49:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:49:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:49:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T13:49:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T13:49:58.690 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-03-09T13:49:58.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:49:58.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T13:49:58.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T13:49:58.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:49:58.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:49:58.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:49:58.717 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:58.716+0000 7fabc640f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:58.719 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:58.718+0000 7fabc640f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:58.721 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:58.720+0000 7fabc640f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:49:58.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T13:49:58.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:49:58.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T13:49:58.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:49:58.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:49:58.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:49:58.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:49:58.974 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:49:58.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:49:58.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:49:59.240 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:49:59.239+0000 7fabc640f640 -1 Falling back to public interface 2024-03-09T13:49:59.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:00.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:00.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:00.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:50:00.248 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:50:00.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:00.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:50:00.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:01.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:01.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:01.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:50:01.533 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:50:01.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:01.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:50:01.639 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:01.627+0000 7fabc640f640 -1 osd.2 77 log_to_monitors true 2024-03-09T13:50:01.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:02.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:02.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:02.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:50:02.886 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:50:02.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:02.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T13:50:03.296 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 82 up_thru 82 down_at 78 last_clean_interval [74,77) [v2:127.0.0.1:6818/2413207643,v1:127.0.0.1:6819/2413207643] [v2:127.0.0.1:6820/2413207643,v1:127.0.0.1:6821/2413207643] exists,up 4da9dd03-8db5-4294-9665-3345b5cecedf 2024-03-09T13:50:03.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:50:03.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:50:03.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:50:03.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:50:03.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:50:03.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:50:03.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:50:03.296 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:50:03.297 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:50:03.297 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:50:03.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:50:03.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:50:03.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:50:03.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:50:03.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:50:03.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:50:03.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:50:03.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:50:03.383 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:50:03.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:50:03.699 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:50:03.699 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:50:03.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:50:03.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:03.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:50:03.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841545 2024-03-09T13:50:03.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841545 2024-03-09T13:50:03.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545' 2024-03-09T13:50:03.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:03.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:50:03.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=249108103180 2024-03-09T13:50:03.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 249108103180 2024-03-09T13:50:03.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545 1-249108103180' 2024-03-09T13:50:03.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:03.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:50:04.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-03-09T13:50:04.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-03-09T13:50:04.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841545 1-249108103180 2-352187318274' 2024-03-09T13:50:04.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:04.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841545 2024-03-09T13:50:04.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:04.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:50:04.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841545 2024-03-09T13:50:04.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:04.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841545 2024-03-09T13:50:04.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841545' 2024-03-09T13:50:04.048 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 283467841545 2024-03-09T13:50:04.048 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:50:04.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841544 -lt 283467841545 2024-03-09T13:50:04.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:50:05.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:50:05.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:50:05.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841545 -lt 283467841545 2024-03-09T13:50:05.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:05.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-249108103180 2024-03-09T13:50:05.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:05.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:50:05.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-249108103180 2024-03-09T13:50:05.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:05.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=249108103180 2024-03-09T13:50:05.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 249108103180' 2024-03-09T13:50:05.608 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 249108103180 2024-03-09T13:50:05.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:50:05.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 249108103181 -lt 249108103180 2024-03-09T13:50:05.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:05.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318274 2024-03-09T13:50:05.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:05.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:50:05.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318274 2024-03-09T13:50:05.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:05.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-03-09T13:50:05.884 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318274' 2024-03-09T13:50:05.884 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 352187318274 2024-03-09T13:50:05.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:50:06.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-03-09T13:50:06.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:50:06.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:50:06.159 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:50:06.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:50:06.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:50:06.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:50:06.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:50:06.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:50:06.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:50:06.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:50:06.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:50:06.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:50:06.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:50:06.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:50:06.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:50:07.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:50:07.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:50:07.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:50:07.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3647: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-09T13:50:07.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:50:07.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ5 2024-03-09T13:50:07.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2024-03-09T13:50:07.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:50:07.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:50:07.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2024-03-09T13:50:07.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:50:07.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:50:07.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:50:07.482 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:50:07.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:50:07.967 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:50:07.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:50:07.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:50:07.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:50:08.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 5 % 2 2024-03-09T13:50:08.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T13:50:08.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:50:08.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3652: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2024-03-09T13:50:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:2+0 records in 2024-03-09T13:50:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:2+0 records out 2024-03-09T13:50:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000189058 s, 21.7 MB/s 2024-03-09T13:50:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3653: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:50:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:50:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:50:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:50:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:50:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:50:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:50:08.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:50:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:50:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:50:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:50:08.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:50:08.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:50:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:50:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:50:08.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:50:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:50:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:50:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:50:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:50:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:50:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:50:08.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:50:08.507 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:50:10.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:50:10.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:50:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:50:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:50:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:50:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:50:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:50:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:50:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:50:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:50:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:50:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:50:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:50:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:50:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:50:10.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:50:10.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:50:10.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:50:10.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:50:10.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:50:10.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:50:10.710 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:50:10.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:50:10.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:50:10.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:50:10.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:50:10.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:50:10.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:50:10.737 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:10.736+0000 7efdcdd9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:50:10.739 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:10.738+0000 7efdcdd9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:50:10.741 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:10.740+0000 7efdcdd9a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:50:10.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:50:10.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:50:10.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:50:10.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:50:10.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:50:10.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:10.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:50:10.995 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:50:10.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:10.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:50:11.258 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:11.257+0000 7efdcdd9a640 -1 Falling back to public interface 2024-03-09T13:50:11.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:12.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:12.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:12.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:50:12.271 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:50:12.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:12.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:50:12.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:13.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:13.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:13.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:50:13.550 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:50:13.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:13.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:50:13.753 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:13.752+0000 7efdcdd9a640 -1 osd.1 87 log_to_monitors true 2024-03-09T13:50:13.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:14.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:14.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:14.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:50:14.838 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:50:14.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:14.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:50:15.147 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 92 up_thru 83 down_at 88 last_clean_interval [58,87) [v2:127.0.0.1:6810/2604023708,v1:127.0.0.1:6811/2604023708] [v2:127.0.0.1:6812/2604023708,v1:127.0.0.1:6813/2604023708] exists,up dd73c514-d245-46ec-bdb1-70ee6e12902e 2024-03-09T13:50:15.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:50:15.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:50:15.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:50:15.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:50:15.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:50:15.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:50:15.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:50:15.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:50:15.148 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:50:15.148 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:50:15.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:50:15.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:50:15.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:50:15.234 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:50:15.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:50:15.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:50:15.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:50:15.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:50:15.235 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:50:15.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:50:15.511 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:50:15.511 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:50:15.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:50:15.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:15.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:50:15.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841548 2024-03-09T13:50:15.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841548 2024-03-09T13:50:15.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548' 2024-03-09T13:50:15.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:15.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:50:15.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991234 2024-03-09T13:50:15.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991234 2024-03-09T13:50:15.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548 1-395136991234' 2024-03-09T13:50:15.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:15.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:50:15.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318277 2024-03-09T13:50:15.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318277 2024-03-09T13:50:15.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-283467841548 1-395136991234 2-352187318277' 2024-03-09T13:50:15.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:15.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-283467841548 2024-03-09T13:50:15.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:15.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:50:15.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-283467841548 2024-03-09T13:50:15.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:15.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841548 2024-03-09T13:50:15.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 283467841548' 2024-03-09T13:50:15.931 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 283467841548 2024-03-09T13:50:15.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:50:16.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841547 -lt 283467841548 2024-03-09T13:50:16.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:50:17.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:50:17.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:50:17.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841547 -lt 283467841548 2024-03-09T13:50:17.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:50:18.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:50:18.486 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:50:18.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841549 -lt 283467841548 2024-03-09T13:50:18.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:18.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-395136991234 2024-03-09T13:50:18.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:18.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:50:18.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-395136991234 2024-03-09T13:50:18.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:18.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991234 2024-03-09T13:50:18.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 395136991234' 2024-03-09T13:50:18.769 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 395136991234 2024-03-09T13:50:18.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:50:19.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991234 -lt 395136991234 2024-03-09T13:50:19.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:19.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318277 2024-03-09T13:50:19.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:19.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:50:19.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318277 2024-03-09T13:50:19.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:19.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318277 2024-03-09T13:50:19.045 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318277' 2024-03-09T13:50:19.045 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 352187318277 2024-03-09T13:50:19.045 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:50:19.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318278 -lt 352187318277 2024-03-09T13:50:19.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:50:19.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:50:19.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:50:19.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:50:19.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:50:19.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:50:19.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:50:19.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:50:19.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:50:19.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:50:19.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:50:19.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:50:19.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:50:19.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:50:19.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:50:20.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:50:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:50:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:50:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:50:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ6 2024-03-09T13:50:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2024-03-09T13:50:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:50:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:50:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2024-03-09T13:50:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:50:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:50:20.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:50:20.676 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:50:20.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:50:21.163 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:50:21.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:50:21.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:50:21.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:50:21.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 6 % 2 2024-03-09T13:50:21.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=0 2024-03-09T13:50:21.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:50:21.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3657: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-09T13:50:21.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:50:21.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:50:21.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:50:21.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:50:21.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-09T13:50:21.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:50:21.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:50:21.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:50:21.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:50:21.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:50:21.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:50:21.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:50:21.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:50:21.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:50:21.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:50:22.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:50:22.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2024-03-09T13:50:22.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:50:22.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:50:22.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:50:22.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:50:22.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:50:22.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2024-03-09T13:50:24.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:50:24.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:50:24.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:50:24.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:50:24.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:50:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:50:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:50:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:50:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:50:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:50:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:50:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:50:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:50:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:50:24.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:50:24.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:50:24.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:50:24.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:50:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:50:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:50:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:50:24.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:50:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:50:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:50:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:50:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:50:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:50:24.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:50:24.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:50:24.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:50:24.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:50:24.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:50:24.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:50:24.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:50:24.472 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:50:24.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:50:24.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:50:24.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:50:24.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:50:24.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:50:24.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:50:24.498 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:24.498+0000 7fb496eaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:50:24.501 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:24.500+0000 7fb496eaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:50:24.502 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:24.502+0000 7fb496eaa640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:50:24.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:50:24.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:50:24.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:50:24.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:50:24.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:50:24.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:24.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:50:24.755 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:50:24.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:24.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:50:25.022 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:25.022+0000 7fb496eaa640 -1 Falling back to public interface 2024-03-09T13:50:25.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:26.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:26.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:26.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:50:26.027 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:50:26.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:26.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:50:26.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:27.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:27.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:27.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:50:27.310 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:50:27.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:27.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:50:27.389 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:27.389+0000 7fb496eaa640 -1 osd.0 96 log_to_monitors true 2024-03-09T13:50:27.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:28.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:28.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:28.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:50:28.653 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:50:28.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:28.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:50:28.939 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 101 up_thru 101 down_at 97 last_clean_interval [66,96) [v2:127.0.0.1:6802/3005392084,v1:127.0.0.1:6803/3005392084] [v2:127.0.0.1:6804/3005392084,v1:127.0.0.1:6805/3005392084] exists,up d7431c32-c2c6-4350-9d07-f7a8f1b761fd 2024-03-09T13:50:28.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:50:28.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:50:28.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:50:28.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:50:28.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:50:28.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:50:28.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:50:28.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:50:28.941 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:50:28.941 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:50:28.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:50:28.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:50:28.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:50:29.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:50:29.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:50:29.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:50:29.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:50:29.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:50:29.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:50:29.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:50:29.301 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:50:29.301 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:50:29.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:50:29.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:29.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:50:29.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696898 2024-03-09T13:50:29.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696898 2024-03-09T13:50:29.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898' 2024-03-09T13:50:29.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:29.404 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:50:29.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=395136991238 2024-03-09T13:50:29.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 395136991238 2024-03-09T13:50:29.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898 1-395136991238' 2024-03-09T13:50:29.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:29.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:50:29.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318281 2024-03-09T13:50:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318281 2024-03-09T13:50:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696898 1-395136991238 2-352187318281' 2024-03-09T13:50:29.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:29.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-433791696898 2024-03-09T13:50:29.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:29.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:50:29.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-433791696898 2024-03-09T13:50:29.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:29.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696898 2024-03-09T13:50:29.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 433791696898' 2024-03-09T13:50:29.615 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 433791696898 2024-03-09T13:50:29.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:50:29.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696898 -lt 433791696898 2024-03-09T13:50:29.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:29.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-395136991238 2024-03-09T13:50:29.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:29.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:50:29.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-395136991238 2024-03-09T13:50:29.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:29.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=395136991238 2024-03-09T13:50:29.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 395136991238' 2024-03-09T13:50:29.895 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 395136991238 2024-03-09T13:50:29.895 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:50:30.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991237 -lt 395136991238 2024-03-09T13:50:30.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:50:31.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:50:31.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:50:31.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991237 -lt 395136991238 2024-03-09T13:50:31.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:50:32.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:50:32.466 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:50:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 395136991238 -lt 395136991238 2024-03-09T13:50:32.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:32.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318281 2024-03-09T13:50:32.743 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:32.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:50:32.744 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318281 2024-03-09T13:50:32.744 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318281 2024-03-09T13:50:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318281' 2024-03-09T13:50:32.746 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 352187318281 2024-03-09T13:50:32.746 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:50:33.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318281 -lt 352187318281 2024-03-09T13:50:33.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:50:33.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:50:33.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:50:33.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:50:33.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:50:33.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:50:33.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:50:33.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:50:33.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:50:33.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:50:33.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:50:33.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:50:33.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:50:33.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:50:33.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:50:34.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:50:34.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:50:34.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:50:34.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3658: corrupt_scrub_erasure: echo -n bad-val 2024-03-09T13:50:34.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3659: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T13:50:34.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:50:34.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:50:34.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:50:34.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:50:34.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T13:50:34.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:50:34.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:50:34.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:50:34.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:50:34.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:50:34.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:50:34.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:50:34.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:50:34.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:50:34.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:50:34.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:50:34.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T13:50:34.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:50:34.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:50:34.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:50:34.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:50:34.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:50:34.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2024-03-09T13:50:36.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:50:36.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:50:36.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:50:36.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:50:36.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:50:36.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:50:36.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:50:36.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:50:36.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:50:36.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:50:36.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:50:36.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:50:36.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:50:36.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:50:36.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:50:36.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:50:36.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:50:36.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:50:36.529 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:50:36.529 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:50:36.529 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:50:36.529 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:50:36.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:50:36.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:50:36.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:50:36.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:50:36.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:50:36.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:50:36.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:50:36.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:50:36.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:50:36.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:50:36.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:50:36.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:50:36.531 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:50:36.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:50:36.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:50:36.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:50:36.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:50:36.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:50:36.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:50:36.556 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:36.556+0000 7f377ccc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:50:36.559 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:36.559+0000 7f377ccc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:50:36.560 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:36.560+0000 7f377ccc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:50:36.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:50:36.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:50:36.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:50:36.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:50:36.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:50:36.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:36.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:50:36.815 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:50:36.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:36.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:50:37.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:37.091 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:37.091+0000 7f377ccc8640 -1 Falling back to public interface 2024-03-09T13:50:38.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:38.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:38.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:50:38.088 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:50:38.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:38.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:50:38.372 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:39.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:39.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:39.374 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:50:39.374 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:50:39.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:39.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:50:39.472 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:39.471+0000 7f377ccc8640 -1 osd.1 103 log_to_monitors true 2024-03-09T13:50:39.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:40.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:40.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:40.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:50:40.661 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:50:40.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:40.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:50:41.113 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 108 up_thru 108 down_at 104 last_clean_interval [92,103) [v2:127.0.0.1:6810/3636047956,v1:127.0.0.1:6811/3636047956] [v2:127.0.0.1:6812/3636047956,v1:127.0.0.1:6813/3636047956] exists,up dd73c514-d245-46ec-bdb1-70ee6e12902e 2024-03-09T13:50:41.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:50:41.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:50:41.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:50:41.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:50:41.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:50:41.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:50:41.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:50:41.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:50:41.114 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:50:41.114 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:50:41.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:50:41.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:50:41.114 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:50:41.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:50:41.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:50:41.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:50:41.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:50:41.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:50:41.201 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:50:41.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:50:41.645 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:50:41.645 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:50:41.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:50:41.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:41.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:50:41.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696901 2024-03-09T13:50:41.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696901 2024-03-09T13:50:41.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901' 2024-03-09T13:50:41.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:41.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:50:41.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467970 2024-03-09T13:50:41.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467970 2024-03-09T13:50:41.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901 1-463856467970' 2024-03-09T13:50:41.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:41.857 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:50:41.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318285 2024-03-09T13:50:41.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318285 2024-03-09T13:50:41.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-433791696901 1-463856467970 2-352187318285' 2024-03-09T13:50:41.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:41.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-433791696901 2024-03-09T13:50:41.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:41.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:50:41.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-433791696901 2024-03-09T13:50:41.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:41.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696901 2024-03-09T13:50:41.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 433791696901' 2024-03-09T13:50:41.964 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 433791696901 2024-03-09T13:50:41.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:50:42.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696900 -lt 433791696901 2024-03-09T13:50:42.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:50:43.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:50:43.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:50:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696900 -lt 433791696901 2024-03-09T13:50:43.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:50:44.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:50:44.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:50:44.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696902 -lt 433791696901 2024-03-09T13:50:44.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:44.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467970 2024-03-09T13:50:44.809 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:44.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:50:44.811 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467970 2024-03-09T13:50:44.811 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:44.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467970 2024-03-09T13:50:44.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467970' 2024-03-09T13:50:44.812 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 463856467970 2024-03-09T13:50:44.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:50:45.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467970 -lt 463856467970 2024-03-09T13:50:45.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:45.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318285 2024-03-09T13:50:45.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:45.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:50:45.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318285 2024-03-09T13:50:45.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:45.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318285 2024-03-09T13:50:45.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318285' 2024-03-09T13:50:45.088 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 352187318285 2024-03-09T13:50:45.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:50:45.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318285 -lt 352187318285 2024-03-09T13:50:45.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:50:45.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:50:45.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:50:45.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:50:45.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:50:45.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:50:45.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:50:45.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:50:45.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:50:45.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:50:45.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:50:45.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:50:45.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:50:45.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:50:45.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:50:46.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:50:46.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:50:46.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:50:46.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3612: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2024-03-09T13:50:46.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3613: corrupt_scrub_erasure: objname=EOBJ7 2024-03-09T13:50:46.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3614: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2024-03-09T13:50:46.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:50:46.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T13:50:46.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2024-03-09T13:50:46.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:50:46.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:50:46.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:50:46.807 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:50:46.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:50:47.192 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:50:47.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:50:47.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:50:47.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:50:47.504 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: expr 7 % 2 2024-03-09T13:50:47.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3616: corrupt_scrub_erasure: local osd=1 2024-03-09T13:50:47.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3618: corrupt_scrub_erasure: case $i in 2024-03-09T13:50:47.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3663: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-09T13:50:47.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3664: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2024-03-09T13:50:47.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3665: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2024-03-09T13:50:47.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3668: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-09T13:50:47.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:50:47.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:50:47.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:50:47.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:50:47.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-09T13:50:47.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:50:47.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:50:47.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:50:47.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:50:47.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:50:47.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:50:47.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:50:47.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:50:47.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:50:47.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:50:47.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:50:47.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2024-03-09T13:50:47.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:50:47.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:50:47.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:50:47.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:50:47.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:50:47.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2024-03-09T13:50:49.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:50:49.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:50:49.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:50:49.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:50:49.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:50:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:50:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:50:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:50:49.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:50:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:50:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:50:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:50:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:50:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:50:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:50:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:50:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:50:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:50:49.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:50:49.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:50:49.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:50:49.502 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:50:49.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:50:49.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:50:49.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:50:49.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:50:49.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:50:49.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:50:49.528 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:49.527+0000 7fc38b019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:50:49.530 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:49.529+0000 7fc38b019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:50:49.531 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:49.530+0000 7fc38b019640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:50:49.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:50:49.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:50:49.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:50:49.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:50:49.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:50:49.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:49.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:50:49.786 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:50:49.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:49.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:50:50.046 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:50.045+0000 7fc38b019640 -1 Falling back to public interface 2024-03-09T13:50:50.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:50:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:50:51.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:51.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:50:51.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:52.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:52.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:52.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:50:52.340 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:50:52.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:52.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:50:52.376 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:50:52.375+0000 7fc38b019640 -1 osd.0 113 log_to_monitors true 2024-03-09T13:50:52.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:50:53.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:50:53.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:50:53.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:50:53.618 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:50:53.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:50:53.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:50:54.014 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 117 up_thru 117 down_at 114 last_clean_interval [101,113) [v2:127.0.0.1:6802/1104580718,v1:127.0.0.1:6803/1104580718] [v2:127.0.0.1:6804/1104580718,v1:127.0.0.1:6805/1104580718] exists,up d7431c32-c2c6-4350-9d07-f7a8f1b761fd 2024-03-09T13:50:54.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:50:54.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:50:54.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:50:54.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:50:54.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:50:54.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:50:54.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:50:54.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:50:54.014 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:50:54.015 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:50:54.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:50:54.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:50:54.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:50:54.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:50:54.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:50:54.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:50:54.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:50:54.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:50:54.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:50:54.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:50:54.453 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:50:54.453 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:50:54.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:50:54.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:54.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:50:54.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=502511173634 2024-03-09T13:50:54.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 502511173634 2024-03-09T13:50:54.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-502511173634' 2024-03-09T13:50:54.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:54.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:50:54.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467973 2024-03-09T13:50:54.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467973 2024-03-09T13:50:54.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-502511173634 1-463856467973' 2024-03-09T13:50:54.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:50:54.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:50:54.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318288 2024-03-09T13:50:54.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318288 2024-03-09T13:50:54.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-502511173634 1-463856467973 2-352187318288' 2024-03-09T13:50:54.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:54.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-502511173634 2024-03-09T13:50:54.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:54.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:50:54.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-502511173634 2024-03-09T13:50:54.768 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:54.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=502511173634 2024-03-09T13:50:54.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 502511173634' 2024-03-09T13:50:54.769 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 502511173634 2024-03-09T13:50:54.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:50:55.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 502511173634 2024-03-09T13:50:55.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:50:56.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:50:56.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:50:56.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 502511173634 -lt 502511173634 2024-03-09T13:50:56.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:56.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467973 2024-03-09T13:50:56.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:56.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:50:56.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467973 2024-03-09T13:50:56.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:56.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467973 2024-03-09T13:50:56.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467973' 2024-03-09T13:50:56.330 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 463856467973 2024-03-09T13:50:56.330 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:50:56.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467974 -lt 463856467973 2024-03-09T13:50:56.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:50:56.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318288 2024-03-09T13:50:56.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:50:56.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:50:56.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318288 2024-03-09T13:50:56.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:50:56.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318288 2024-03-09T13:50:56.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318288' 2024-03-09T13:50:56.608 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 352187318288 2024-03-09T13:50:56.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:50:56.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318288 -lt 352187318288 2024-03-09T13:50:56.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:50:56.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:50:56.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:50:57.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:50:57.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:50:57.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:50:57.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:50:57.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:50:57.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:50:57.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:50:57.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:50:57.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:50:57.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:50:57.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:50:57.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:50:57.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:50:57.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:50:57.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:50:57.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3669: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T13:50:57.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:50:57.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:50:57.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:50:57.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:50:57.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T13:50:57.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:50:57.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:50:57.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:50:57.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:50:57.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:50:57.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:50:57.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:50:57.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:50:57.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:50:57.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:50:58.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:50:58.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T13:50:58.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:50:58.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:50:58.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:50:58.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:50:58.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:50:58.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2024-03-09T13:51:00.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:51:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:51:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:51:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:51:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:51:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:51:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:51:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:51:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:51:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:51:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:51:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:51:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:51:00.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:51:00.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:51:00.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:51:00.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:51:00.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:51:00.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:51:00.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:51:00.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:51:00.747 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:51:00.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:51:00.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:51:00.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:51:00.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:51:00.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:51:00.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:51:00.775 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:00.774+0000 7fc345573640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:51:00.777 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:00.776+0000 7fc345573640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:51:00.779 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:00.778+0000 7fc345573640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:51:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:51:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:51:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:51:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:51:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:51:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:51:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:51:01.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:01.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:01.293 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:01.292+0000 7fc345573640 -1 Falling back to public interface 2024-03-09T13:51:01.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:02.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:02.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:02.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:51:02.304 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:51:02.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:02.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:02.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:03.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:03.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:03.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:51:03.577 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:51:03.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:03.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:03.728 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:03.727+0000 7fc345573640 -1 osd.0 119 log_to_monitors true 2024-03-09T13:51:03.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:04.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:04.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:04.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:51:04.860 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:51:04.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:04.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:05.197 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 124 up_thru 117 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6802/1904726667,v1:127.0.0.1:6803/1904726667] [v2:127.0.0.1:6804/1904726667,v1:127.0.0.1:6805/1904726667] exists,up d7431c32-c2c6-4350-9d07-f7a8f1b761fd 2024-03-09T13:51:05.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:51:05.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:51:05.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:51:05.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:51:05.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:51:05.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:51:05.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:51:05.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:51:05.198 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:51:05.198 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:51:05.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:51:05.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:51:05.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:51:05.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:51:05.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:51:05.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:51:05.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:51:05.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:51:05.286 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:51:05.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:51:05.626 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:51:05.626 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T13:51:05.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:51:05.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:51:05.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:51:05.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=532575944706 2024-03-09T13:51:05.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 532575944706 2024-03-09T13:51:05.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-532575944706' 2024-03-09T13:51:05.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:51:05.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:51:05.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=463856467977 2024-03-09T13:51:05.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 463856467977 2024-03-09T13:51:05.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-532575944706 1-463856467977' 2024-03-09T13:51:05.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:51:05.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T13:51:06.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318291 2024-03-09T13:51:06.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318291 2024-03-09T13:51:06.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-532575944706 1-463856467977 2-352187318291' 2024-03-09T13:51:06.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:51:06.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-532575944706 2024-03-09T13:51:06.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:51:06.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:51:06.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-532575944706 2024-03-09T13:51:06.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:51:06.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=532575944706 2024-03-09T13:51:06.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 532575944706' 2024-03-09T13:51:06.105 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 532575944706 2024-03-09T13:51:06.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:51:06.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 532575944706 2024-03-09T13:51:06.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:51:07.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:51:07.379 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:51:07.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 532575944706 -lt 532575944706 2024-03-09T13:51:07.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:51:07.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-463856467977 2024-03-09T13:51:07.672 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:51:07.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:51:07.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-463856467977 2024-03-09T13:51:07.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:51:07.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=463856467977 2024-03-09T13:51:07.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 463856467977' 2024-03-09T13:51:07.674 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 463856467977 2024-03-09T13:51:07.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:51:07.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 463856467977 -lt 463856467977 2024-03-09T13:51:07.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:51:07.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-352187318291 2024-03-09T13:51:07.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:51:07.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T13:51:07.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-352187318291 2024-03-09T13:51:07.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:51:07.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318291 2024-03-09T13:51:07.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 352187318291' 2024-03-09T13:51:07.956 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.2 seq 352187318291 2024-03-09T13:51:07.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T13:51:08.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318292 -lt 352187318291 2024-03-09T13:51:08.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:51:08.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:51:08.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:51:08.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:51:08.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:51:08.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:51:08.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:51:08.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:51:08.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:51:08.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:51:08.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:51:08.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:51:08.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:51:08.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:51:08.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:51:09.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-09T13:51:09.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-03-09T13:51:09.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:51:09.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-09T13:51:09.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:51:09.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:51:09.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:51:09.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:51:09.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:51:09.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:51:09.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:51:09.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:51:09.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:51:09.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:51:09.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:51:09.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:51:09.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:51:09.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:51:09.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:51:09.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:51:09.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:51:09.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3670: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2024-03-09T13:51:09.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2024-03-09T13:51:09.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T13:51:09.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=EOBJ0 2024-03-09T13:51:09.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool EOBJ0 2024-03-09T13:51:09.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:51:10.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3676: corrupt_scrub_erasure: local pg=3.0 2024-03-09T13:51:10.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3678: corrupt_scrub_erasure: pg_scrub 3.0 2024-03-09T13:51:10.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-09T13:51:10.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-09T13:51:10.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:51:10.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:51:10.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:51:10.210 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:51:10.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:10.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-09T13:51:10.753 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 3.0s0 on osd.1 to scrub 2024-03-09T13:51:10.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:10.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T13:51:10.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:10.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:51:10.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:51:10.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:51:10.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:51:10.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:51:10.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:51:10.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:51:10.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:51:11.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:48:43.754798+0000 '>' 2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:11.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:51:12.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:51:12.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:51:12.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:51:12.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:51:12.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:51:12.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:51:12.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:51:12.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:48:43.754798+0000 '>' 2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:12.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:51:13.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:51:13.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:51:13.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:51:13.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:51:13.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:51:13.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:51:13.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:51:13.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:48:43.754798+0000 '>' 2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:13.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:51:14.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:51:14.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:51:14.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:51:14.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:51:14.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:51:14.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:51:14.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:51:14.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:48:43.754798+0000 '>' 2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:14.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:51:15.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:51:15.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:51:15.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:51:15.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:51:15.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:51:15.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:51:15.884 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:51:16.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:51:11.436067+0000 '>' 2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:16.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:51:16.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3680: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-09T13:51:16.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-09T13:51:16.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3682: corrupt_scrub_erasure: test 1 = 1 2024-03-09T13:51:16.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T13:51:16.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3684: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-09T13:51:16.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3686: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-09T13:51:16.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-09T13:51:16.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3688: corrupt_scrub_erasure: epoch=124 2024-03-09T13:51:16.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T13:51:16.220 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T13:51:16.220 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T13:51:16.220 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T13:51:16.221 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T13:51:16.221 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T13:51:16.221 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T13:51:16.221 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T13:51:16.221 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T13:51:16.221 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T13:51:16.221 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T13:51:16.221 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T13:51:16.221 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T13:51:16.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T13:51:16.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3690: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T13:51:16.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T13:51:16.244 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T13:51:16.244 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T13:51:16.244 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T13:51:16.244 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T13:51:16.244 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T13:51:16.244 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T13:51:16.244 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T13:51:16.245 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T13:51:16.245 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T13:51:16.245 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T13:51:16.245 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T13:51:16.245 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T13:51:16.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T13:51:16.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4310: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T13:51:16.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4311: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T13:51:16.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T13:51:16.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4312: corrupt_scrub_erasure: test no = yes 2024-03-09T13:51:16.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4317: corrupt_scrub_erasure: test '' = yes 2024-03-09T13:51:16.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4322: corrupt_scrub_erasure: pg_deep_scrub 3.0 2024-03-09T13:51:16.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-09T13:51:16.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:51:16.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:51:16.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:51:16.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:51:16.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:51:16.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:16.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-09T13:51:16.819 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2024-03-09T13:51:16.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-09T13:48:43.754798+0000 last_deep_scrub_stamp 2024-03-09T13:51:16.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T13:51:16.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:16.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-09T13:51:16.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:51:16.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:51:16.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:51:16.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:51:16.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:51:16.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:51:16.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:51:17.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:48:43.754798+0000 '>' 2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:17.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:51:18.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:51:18.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:51:18.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:51:18.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:51:18.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:51:18.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:51:18.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:51:18.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:48:43.754798+0000 '>' 2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:18.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:51:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:51:19.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:51:19.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:51:19.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:51:19.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:51:19.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:51:19.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:51:19.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:48:43.754798+0000 '>' 2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:19.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:51:20.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:51:20.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:51:20.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:51:20.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:51:20.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:51:20.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:51:20.675 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:51:20.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:48:43.754798+0000 '>' 2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:20.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:51:21.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:51:21.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:51:21.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:51:21.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:51:21.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:51:21.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:51:21.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:51:22.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:51:17.671179+0000 '>' 2024-03-09T13:48:43.754798+0000 2024-03-09T13:51:22.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:51:22.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4324: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2024-03-09T13:51:22.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2024-03-09T13:51:22.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4326: corrupt_scrub_erasure: test 1 = 1 2024-03-09T13:51:22.258 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T13:51:22.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4328: corrupt_scrub_erasure: test 3.0 = 3.0 2024-03-09T13:51:22.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4330: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2024-03-09T13:51:22.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2024-03-09T13:51:22.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4332: corrupt_scrub_erasure: epoch=124 2024-03-09T13:51:22.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4334: corrupt_scrub_erasure: '[' true = true ']' 2024-03-09T13:51:22.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T13:51:22.291 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T13:51:22.291 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T13:51:22.291 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T13:51:22.291 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T13:51:22.291 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T13:51:22.291 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T13:51:22.291 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T13:51:22.291 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T13:51:22.291 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T13:51:22.291 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T13:51:22.292 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T13:51:22.292 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T13:51:22.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T13:51:22.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4336: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T13:51:22.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq 'def walk(f): 2024-03-09T13:51:22.314 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T13:51:22.314 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T13:51:22.314 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T13:51:22.314 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T13:51:22.315 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T13:51:22.315 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T13:51:22.315 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T13:51:22.315 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T13:51:22.315 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T13:51:22.315 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T13:51:22.315 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T13:51:22.315 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T13:51:22.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: jq .inconsistents 2024-03-09T13:51:22.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5724: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T13:51:22.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5725: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T13:51:22.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T13:51:22.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5726: corrupt_scrub_erasure: test no = yes 2024-03-09T13:51:22.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5737: corrupt_scrub_erasure: test '' = yes 2024-03-09T13:51:22.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5742: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2024-03-09T13:51:22.706 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' removed 2024-03-09T13:51:22.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:51:22.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:51:22.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:51:22.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:51:22.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:51:22.719 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:51:22.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:51:22.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:51:22.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:51:22.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:51:22.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:51:22.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:51:22.829 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:51:22.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:51:22.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:51:22.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:51:22.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:51:22.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:51:22.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:51:22.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:51:22.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:51:22.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:51:22.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:51:22.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:51:22.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:51:22.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:51:22.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:51:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:51:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:51:22.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:51:22.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:51:22.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:51:22.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:51:22.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:51:22.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:51:22.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:51:22.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:51:22.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:51:22.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:51:22.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:51:22.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:51:22.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:51:22.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:51:22.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:51:22.866 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:51:22.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:51:22.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:51:22.867 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:51:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:51:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:51:22.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:51:22.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:51:22.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:51:22.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:51:22.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:51:22.872 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:51:22.872 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:51:22.872 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:51:22.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:51:22.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:51:22.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:51:22.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:51:22.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:51:22.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:51:22.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:51:22.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:51:22.876 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:51:22.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:51:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:51:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:51:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2024-03-09T13:51:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2024-03-09T13:51:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:992: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2024-03-09T13:51:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:993: TEST_corrupt_scrub_replicated: local total_objs=19 2024-03-09T13:51:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-09T13:51:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:51:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:51:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:51:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:51:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:51:22.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-09T13:51:23.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:51:23.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:51:23.339 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:51:23.339 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:51:23.340 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:51:23.340 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:51:23.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:51:23.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-09T13:51:23.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:51:23.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:51:23.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:51:23.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:51:23.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:51:23.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:51:23.533 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:51:23.533 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:51:23.533 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:51:23.534 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:51:23.534 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:51:23.534 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:51:23.534 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:51:23.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:51:23.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:51:23.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:51:23.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:51:23.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:51:23.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:51:23.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:51:23.621 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:51:23.621 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:51:23.621 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:51:23.621 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:51:23.621 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:51:23.621 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:51:23.621 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:51:23.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:51:23.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:51:23.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:996: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-03-09T13:51:23.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:51:23.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:51:23.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:51:23.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:51:23.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:51:23.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:51:24.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:51:24.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:51:24.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:51:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:51:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:51:24.508 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:51:24.508 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:51:24.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:51:24.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:51:24.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:997: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2024-03-09T13:51:24.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:51:24.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:51:24.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:51:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:51:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:51:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:51:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:51:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:51:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:51:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:51:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:51:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:51:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:51:24.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:51:24.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:51:24.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:51:24.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:51:24.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:51:24.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:51:24.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:51:24.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:51:24.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:51:24.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:51:24.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f88b04f5-8017-47a7-91b8-f6d2e01cd1ea' 2024-03-09T13:51:24.543 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:51:24.543 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:51:24.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBcaexlqKU4IRAAn8cNWXaI/BgJaXm2mHeHKQ== 2024-03-09T13:51:24.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBcaexlqKU4IRAAn8cNWXaI/BgJaXm2mHeHKQ=="}' 2024-03-09T13:51:24.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f88b04f5-8017-47a7-91b8-f6d2e01cd1ea -i td/osd-scrub-repair/0/new.json 2024-03-09T13:51:24.764 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:51:24.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:51:24.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcaexlqKU4IRAAn8cNWXaI/BgJaXm2mHeHKQ== --osd-uuid f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:51:24.804 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:24.803+0000 7f8300e68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:51:24.815 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:24.815+0000 7f8300e68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:51:24.816 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:24.816+0000 7f8300e68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:51:24.816 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:24.816+0000 7f8300e68640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:51:24.816 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:24.816+0000 7f8300e68640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:51:24.817 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:24.816+0000 7f8300e68640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:51:24.907 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:24.906+0000 7f8300e68640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:51:30.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:51:30.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:51:30.613 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:51:30.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:51:30.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:51:30.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:51:30.973 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:51:30.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:51:30.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:51:30.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:51:30.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:51:30.997 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:30.997+0000 7faef4c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:51:30.999 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:30.999+0000 7faef4c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:51:31.000 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:31.000+0000 7faef4c0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:51:31.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:51:31.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:51:31.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:51:31.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:51:31.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:51:31.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:31.246 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:51:31.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:51:31.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:31.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:31.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:31.536 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:31.536+0000 7faef4c0d640 -1 Falling back to public interface 2024-03-09T13:51:32.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:32.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:32.521 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:51:32.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:51:32.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:32.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:32.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:33.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:33.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:33.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:51:33.801 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:51:33.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:33.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:33.862 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:33.861+0000 7faef4c0d640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:51:34.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:35.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:35.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:35.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:51:35.131 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:51:35.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:35.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:35.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:36.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:36.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:36.451 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:51:36.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:51:36.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:36.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:36.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:37.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:37.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:37.836 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:51:37.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:51:37.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:37.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:38.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:39.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:39.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:39.117 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:51:39.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:51:39.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:39.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:39.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:40.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:40.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:40.410 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:51:40.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:51:40.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:40.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:40.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:41.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:41.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:41.818 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:51:41.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:51:41.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:41.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:42.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:43.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:43.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:43.105 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:51:43.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:51:43.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:43.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:43.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:44.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:44.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:44.432 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:51:44.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:51:44.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:44.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:44.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:45.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:45.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:45.791 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:51:45.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:51:45.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:45.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:46.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:47.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:47.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:47.528 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:51:47.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:51:47.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:47.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:48.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:49.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:49.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:49.043 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:51:49.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:51:49.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:49.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:49.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:50.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:50.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:50.787 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:51:50.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:51:50.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:50.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:51.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:52.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:52.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:52.489 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:51:52.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:51:52.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:52.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:52.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:53.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:53.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:53.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:51:53.881 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:51:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:53.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:54.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:55.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:55.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:55.594 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:51:55.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:51:55.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:55.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:56.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:51:57.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:51:57.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:51:57.040 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:51:57.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:51:57.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:51:57.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:51:57.313 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2145846736,v1:127.0.0.1:6803/2145846736] [v2:127.0.0.1:6804/2145846736,v1:127.0.0.1:6805/2145846736] exists,up f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:51:57.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:51:57.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:51:57.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:51:57.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2024-03-09T13:51:57.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:51:57.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:51:57.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:51:57.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:51:57.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:51:57.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:51:57.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:51:57.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:51:57.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:51:57.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:51:57.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:51:57.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:51:57.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:51:57.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:51:57.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:51:57.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:51:57.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:51:57.320 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:51:57.320 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:51:57.320 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:51:57.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:51:57.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:51:57.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:51:57.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:51:57.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:51:57.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:51:57.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:51:57.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:51:57.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:51:57.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:51:57.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:51:57.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:51:57.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:51:57.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:51:57.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:51:57.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:51:57.322 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:51:57.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c' 2024-03-09T13:51:57.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:51:57.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9aexlf7jyExAA0y9TMC4lIfxTgfdUXpLFMA== 2024-03-09T13:51:57.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9aexlf7jyExAA0y9TMC4lIfxTgfdUXpLFMA=="}' 2024-03-09T13:51:57.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c -i td/osd-scrub-repair/1/new.json 2024-03-09T13:51:57.663 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:51:57.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:51:57.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9aexlf7jyExAA0y9TMC4lIfxTgfdUXpLFMA== --osd-uuid 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:51:57.702 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:57.702+0000 7f108b211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:51:57.764 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:57.764+0000 7f108b211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:51:57.765 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:57.765+0000 7f108b211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:51:57.765 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:57.765+0000 7f108b211640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:51:57.766 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:57.765+0000 7f108b211640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:51:57.766 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:57.765+0000 7f108b211640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:51:57.839 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:51:57.839+0000 7f108b211640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:52:03.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:52:03.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:52:03.893 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:52:03.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:52:03.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:52:04.246 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:52:04.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:52:04.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:52:04.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:52:04.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:52:04.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:52:04.272 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:52:04.272+0000 7f18be404640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:52:04.274 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:52:04.274+0000 7f18be404640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:52:04.275 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:52:04.275+0000 7f18be404640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:52:04.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:52:04.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:52:04.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:52:04.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:52:04.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:52:04.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:04.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:52:04.520 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:52:04.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:04.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:04.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:04.798 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:52:04.798+0000 7f18be404640 -1 Falling back to public interface 2024-03-09T13:52:05.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:05.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:05.792 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:52:05.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:52:05.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:05.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:06.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:07.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:07.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:07.066 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:52:07.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:52:07.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:07.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:07.167 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:52:07.166+0000 7f18be404640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:52:07.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:08.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:08.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:08.377 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:52:08.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:52:08.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:08.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:08.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:09.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:09.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:09.659 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:52:09.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:52:09.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:09.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:09.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:10.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:11.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:11.000 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:52:11.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:52:11.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:11.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:11.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:12.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:12.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:12.385 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:52:12.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:52:12.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:12.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:12.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:13.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:13.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:13.668 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:52:13.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:52:13.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:13.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:14.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:15.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:15.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:15.027 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:52:15.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:52:15.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:15.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:15.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:16.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:16.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:16.310 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:52:16.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:52:16.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:16.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:16.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:17.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:17.639 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:52:17.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:17.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:52:17.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:17.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:18.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:19.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:19.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:19.041 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:52:19.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:52:19.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:19.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:19.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:20.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:20.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:20.329 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:52:20.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:52:20.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:20.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:20.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:21.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:21.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:21.922 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:52:21.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:52:21.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:21.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:22.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:23.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:23.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:23.666 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:52:23.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:52:23.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:23.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:24.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:25.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:25.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:25.184 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:52:25.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:52:25.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:25.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:25.976 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:26.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:26.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:26.978 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:52:26.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:52:26.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:26.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:27.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:28.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:28.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:28.364 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:52:28.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:52:28.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:28.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:28.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:29.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:29.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:29.958 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:52:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:52:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:30.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:52:31.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:52:31.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:52:31.245 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:52:31.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:52:31.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:52:31.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:52:31.526 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3255709153,v1:127.0.0.1:6811/3255709153] [v2:127.0.0.1:6812/3255709153,v1:127.0.0.1:6813/3255709153] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:52:31.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:52:31.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:52:31.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:52:31.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: TEST_corrupt_scrub_replicated: create_rbd_pool 2024-03-09T13:52:31.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T13:52:31.792 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T13:52:31.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T13:52:31.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T13:52:32.185 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T13:52:32.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:52:33.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T13:52:34.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1000: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-09T13:52:34.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:52:34.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:52:34.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:52:34.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:52:34.392 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:52:34.392 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:52:34.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:52:34.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:52:34.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:52:34.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:52:34.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:52:34.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:52:34.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:52:34.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:52:34.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:52:34.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:52:34.789 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:52:34.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:52:34.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:52:34.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:52:34.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836493 2024-03-09T13:52:34.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836493 2024-03-09T13:52:34.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493' 2024-03-09T13:52:34.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:52:34.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:52:34.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-03-09T13:52:34.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-03-09T13:52:34.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836493 1-42949672965' 2024-03-09T13:52:34.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:52:34.998 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836493 2024-03-09T13:52:34.998 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:52:34.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:52:34.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836493 2024-03-09T13:52:34.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:52:35.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836493 2024-03-09T13:52:35.000 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836493 2024-03-09T13:52:35.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-03-09T13:52:35.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:52:35.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-03-09T13:52:35.273 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:52:36.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:52:36.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:52:36.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836492 -lt 21474836493 2024-03-09T13:52:36.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:52:37.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:52:37.560 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:52:37.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836493 2024-03-09T13:52:37.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:52:37.831 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-03-09T13:52:37.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:52:37.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:52:37.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-03-09T13:52:37.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:52:37.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-03-09T13:52:37.835 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672965 2024-03-09T13:52:37.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-03-09T13:52:37.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:52:38.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672965 2024-03-09T13:52:38.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:52:38.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:52:38.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:52:38.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T13:52:38.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:52:38.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:52:38.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:52:38.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:52:38.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:52:38.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:52:38.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:52:38.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:52:38.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:52:38.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:52:38.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:52:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T13:52:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:52:39.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:52:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: TEST_corrupt_scrub_replicated: create_pool foo 1 2024-03-09T13:52:39.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-09T13:52:39.620 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'foo' created 2024-03-09T13:52:39.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:52:40.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2024-03-09T13:52:40.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-03-09T13:52:41.068 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'csr_pool' created 2024-03-09T13:52:41.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:52:42.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1004: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-09T13:52:42.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:52:42.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:52:42.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:52:42.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:52:42.082 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:52:42.082 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:52:42.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:52:42.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:52:42.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:52:42.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:52:42.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:52:42.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:52:42.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:52:42.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:52:42.182 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:52:42.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:52:42.453 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:52:42.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:52:42.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:52:42.454 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:52:42.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-03-09T13:52:42.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-03-09T13:52:42.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-03-09T13:52:42.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:52:42.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:52:42.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-03-09T13:52:42.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-03-09T13:52:42.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672968' 2024-03-09T13:52:42.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:52:42.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-03-09T13:52:42.668 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:52:42.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:52:42.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-03-09T13:52:42.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:52:42.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-03-09T13:52:42.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-03-09T13:52:42.671 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836496 2024-03-09T13:52:42.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:52:42.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836496 2024-03-09T13:52:42.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:52:43.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:52:43.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:52:44.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836495 -lt 21474836496 2024-03-09T13:52:44.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:52:45.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:52:45.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:52:45.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-03-09T13:52:45.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:52:45.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-03-09T13:52:45.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:52:45.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:52:45.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-03-09T13:52:45.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:52:45.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-03-09T13:52:45.526 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672968 2024-03-09T13:52:45.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-03-09T13:52:45.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:52:45.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-03-09T13:52:45.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:52:45.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:52:45.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:52:46.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:52:46.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:52:46.138 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:52:46.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:52:46.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:52:46.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:52:46.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:52:46.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:52:46.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:52:46.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:52:46.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:52:46.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:52:46.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:52:46.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:52:46.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:52:46.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: seq 1 19 2024-03-09T13:52:46.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:52:46.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-03-09T13:52:46.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-03-09T13:52:46.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:52:46.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:52:46.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-03-09T13:52:46.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:52:46.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:52:46.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:52:47.153 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:52:47.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:52:47.663 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:52:47.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:52:47.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:52:47.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:52:47.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-03-09T13:52:48.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-03-09T13:52:48.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:52:48.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-03-09T13:52:48.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-03-09T13:52:48.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:52:48.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:52:48.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-03-09T13:52:48.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:52:48.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:52:48.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:52:48.482 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:52:48.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:52:48.993 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:52:49.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:52:49.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:52:49.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:52:49.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-03-09T13:52:49.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-03-09T13:52:49.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:52:49.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-03-09T13:52:49.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2024-03-09T13:52:49.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:52:49.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:52:49.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2024-03-09T13:52:49.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:52:49.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:52:49.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:52:49.912 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:52:49.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:52:50.289 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:52:50.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:52:50.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:52:50.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:52:50.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2024-03-09T13:52:50.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2024-03-09T13:52:50.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:52:50.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-03-09T13:52:50.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2024-03-09T13:52:50.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:52:50.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:52:50.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2024-03-09T13:52:50.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:52:50.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:52:50.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:52:51.217 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:52:51.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:52:51.677 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:52:51.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:52:51.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:52:51.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:52:51.905 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2024-03-09T13:52:52.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2024-03-09T13:52:52.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:52:52.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-03-09T13:52:52.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2024-03-09T13:52:52.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:52:52.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:52:52.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2024-03-09T13:52:52.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:52:52.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:52:52.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:52:52.538 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:52:52.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:52:53.107 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:52:53.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:52:53.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:52:53.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:52:53.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2024-03-09T13:52:53.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2024-03-09T13:52:53.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:52:53.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-03-09T13:52:53.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2024-03-09T13:52:53.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:52:53.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:52:53.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2024-03-09T13:52:53.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:52:53.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:52:53.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:52:53.968 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:52:53.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:52:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:52:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:52:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:52:54.515 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:52:54.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2024-03-09T13:52:54.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2024-03-09T13:52:54.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:52:54.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-03-09T13:52:54.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2024-03-09T13:52:54.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:52:54.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:52:54.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2024-03-09T13:52:54.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:52:54.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:52:54.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:52:55.331 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:52:55.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:52:55.917 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:52:55.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:52:55.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:52:55.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:52:56.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2024-03-09T13:52:56.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2024-03-09T13:52:56.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:52:56.413 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-03-09T13:52:56.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2024-03-09T13:52:56.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:52:56.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:52:56.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2024-03-09T13:52:56.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:52:56.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:52:56.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:52:56.913 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:52:56.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:52:57.322 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:52:57.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:52:57.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:52:57.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:52:57.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2024-03-09T13:52:57.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2024-03-09T13:52:57.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:52:57.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-09T13:52:57.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2024-03-09T13:52:57.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:52:57.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:52:57.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2024-03-09T13:52:57.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:52:57.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:52:57.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:52:58.208 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:52:58.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:52:58.769 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:52:58.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:52:58.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:52:58.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:52:59.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2024-03-09T13:52:59.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2024-03-09T13:52:59.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:52:59.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-09T13:52:59.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2024-03-09T13:52:59.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:52:59.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:52:59.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2024-03-09T13:52:59.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:52:59.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:52:59.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:52:59.621 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:52:59.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:53:00.264 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:53:00.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:53:00.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:53:00.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:53:00.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2024-03-09T13:53:00.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2024-03-09T13:53:00.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:00.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-03-09T13:53:00.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2024-03-09T13:53:00.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:53:00.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:53:00.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2024-03-09T13:53:00.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:53:00.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:53:00.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:53:01.225 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:53:01.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:53:01.752 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:53:01.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:53:01.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:53:01.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:53:01.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2024-03-09T13:53:02.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2024-03-09T13:53:02.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:02.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-03-09T13:53:02.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2024-03-09T13:53:02.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:53:02.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:53:02.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2024-03-09T13:53:02.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:53:02.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:53:02.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:53:02.672 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:53:02.683 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:53:03.082 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:53:03.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:53:03.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:53:03.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:53:03.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2024-03-09T13:53:03.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2024-03-09T13:53:03.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:03.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-03-09T13:53:03.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2024-03-09T13:53:03.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:53:03.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:53:03.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2024-03-09T13:53:03.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:53:03.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:53:03.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:53:04.001 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:53:04.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:53:04.596 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:53:04.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:53:04.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:53:04.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:53:04.816 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2024-03-09T13:53:05.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2024-03-09T13:53:05.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:05.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-03-09T13:53:05.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2024-03-09T13:53:05.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:53:05.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:53:05.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2024-03-09T13:53:05.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:53:05.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:53:05.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:53:05.515 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:53:05.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:53:06.034 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:53:06.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:53:06.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:53:06.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:53:06.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2024-03-09T13:53:06.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2024-03-09T13:53:06.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:06.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-03-09T13:53:06.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2024-03-09T13:53:06.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:53:06.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:53:06.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2024-03-09T13:53:06.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:53:06.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:53:06.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:53:06.913 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:53:06.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:53:07.490 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:53:07.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:53:07.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:53:07.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:53:07.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2024-03-09T13:53:07.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2024-03-09T13:53:07.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:07.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-03-09T13:53:07.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2024-03-09T13:53:07.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:53:07.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:53:07.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2024-03-09T13:53:07.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:53:07.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:53:07.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:53:08.351 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:53:08.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:53:08.786 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:53:08.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:53:08.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:53:08.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:53:09.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2024-03-09T13:53:09.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2024-03-09T13:53:09.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:09.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-03-09T13:53:09.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2024-03-09T13:53:09.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:53:09.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:53:09.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2024-03-09T13:53:09.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:53:09.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:53:09.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:53:09.698 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:53:09.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:53:10.082 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:53:10.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:53:10.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:53:10.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:53:10.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2024-03-09T13:53:10.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2024-03-09T13:53:10.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:10.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-03-09T13:53:10.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2024-03-09T13:53:10.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:53:10.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:53:10.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2024-03-09T13:53:10.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:53:10.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:53:10.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:53:11.011 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:53:11.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:53:11.447 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:53:11.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:53:11.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:53:11.458 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:53:11.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2024-03-09T13:53:11.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2024-03-09T13:53:11.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:11.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1007: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-03-09T13:53:11.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1008: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2024-03-09T13:53:11.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:53:11.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:53:11.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2024-03-09T13:53:11.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:53:11.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:53:11.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:53:12.366 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:53:12.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:53:12.793 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:53:12.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:53:12.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:53:12.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:53:13.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2024-03-09T13:53:13.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1011: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2024-03-09T13:53:13.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1015: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2024-03-09T13:53:13.401 INFO:tasks.workunit.client.0.smithi162.stderr:1025+0 records in 2024-03-09T13:53:13.401 INFO:tasks.workunit.client.0.smithi162.stderr:1025+0 records out 2024-03-09T13:53:13.401 INFO:tasks.workunit.client.0.smithi162.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00189429 s, 554 MB/s 2024-03-09T13:53:13.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2024-03-09T13:53:13.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2024-03-09T13:53:13.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-03-09T13:53:13.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-09T13:53:13.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-09T13:53:13.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-09T13:53:13.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:53:13.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: TEST_corrupt_scrub_replicated: local pg=3.0 2024-03-09T13:53:13.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2024-03-09T13:53:13.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-03-09T13:53:13.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-03-09T13:53:13.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-03-09T13:53:13.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:53:14.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: TEST_corrupt_scrub_replicated: local primary=1 2024-03-09T13:53:14.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2024-03-09T13:53:14.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T13:53:14.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T13:53:14.120 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:53:14.120 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:53:14.120 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:53:14.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.0.asok 2024-03-09T13:53:14.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-03-09T13:53:14.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.70042/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2024-03-09T13:53:14.197 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T13:53:14.198 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-03-09T13:53:14.198 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T13:53:14.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2024-03-09T13:53:14.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T13:53:14.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T13:53:14.206 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:53:14.206 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:53:14.206 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:53:14.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.1.asok 2024-03-09T13:53:14.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2024-03-09T13:53:14.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1025: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.70042/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2024-03-09T13:53:14.283 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T13:53:14.284 INFO:tasks.workunit.client.0.smithi162.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2024-03-09T13:53:14.284 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T13:53:14.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-03-09T13:53:14.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-09T13:53:14.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:53:14.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:53:14.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:53:14.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:53:14.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:53:14.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-09T13:52:41.000680+0000 2024-03-09T13:53:14.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-09T13:53:14.828 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-03-09T13:53:14.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-09T13:52:41.000680+0000 last_deep_scrub_stamp 2024-03-09T13:53:14.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T13:53:14.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:52:41.000680+0000 2024-03-09T13:53:14.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-09T13:53:14.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:53:14.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:53:14.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:53:14.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:53:14.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:53:14.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:53:14.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:53:15.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:52:41.000680+0000 '>' 2024-03-09T13:52:41.000680+0000 2024-03-09T13:53:15.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:53:16.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:53:16.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:53:16.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:53:16.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:53:16.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:53:16.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:53:16.111 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:53:16.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:52:41.000680+0000 '>' 2024-03-09T13:52:41.000680+0000 2024-03-09T13:53:16.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:53:17.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:53:17.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:53:17.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:53:17.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:53:17.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:53:17.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:53:17.392 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:53:17.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:52:41.000680+0000 '>' 2024-03-09T13:52:41.000680+0000 2024-03-09T13:53:17.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:53:18.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:53:18.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:53:18.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:53:18.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:53:18.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:53:18.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:53:18.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:53:18.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:53:15.768738+0000 '>' 2024-03-09T13:52:41.000680+0000 2024-03-09T13:53:18.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:53:18.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: seq 1 19 2024-03-09T13:53:18.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:18.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ1 2024-03-09T13:53:18.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 1 % 2 2024-03-09T13:53:18.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T13:53:18.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:53:18.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1038: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2024-03-09T13:53:18.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1039: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2024-03-09T13:53:18.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1040: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:53:18.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:53:18.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:53:18.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:53:18.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:53:18.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:53:18.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:53:18.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:53:18.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:53:18.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:53:18.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:53:18.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:53:18.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:53:18.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:53:18.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:53:18.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:53:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:53:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:53:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:53:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:53:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:53:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:53:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:53:19.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:53:20.536 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2024-03-09T13:53:21.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:53:21.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:53:21.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:53:21.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:53:21.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:53:21.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:53:21.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:53:21.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:53:21.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:53:21.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:53:21.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:53:21.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:53:21.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:53:21.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:53:21.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:53:21.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:53:21.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:53:21.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:53:21.089 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:53:21.089 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:53:21.089 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:53:21.089 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:53:21.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:53:21.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:53:21.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:53:21.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:53:21.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:53:21.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:53:21.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:53:21.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:53:21.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:53:21.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:53:21.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:53:21.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:53:21.091 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:53:21.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:53:21.091 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:53:21.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:53:21.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:53:21.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:53:21.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:53:21.116 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:21.115+0000 7fe6d681a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:53:21.118 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:21.117+0000 7fe6d681a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:53:21.119 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:21.118+0000 7fe6d681a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:53:21.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:53:21.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:53:21.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:53:21.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:53:21.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:53:21.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:21.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:53:21.371 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:53:21.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:21.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:53:21.634 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:21.633+0000 7fe6d681a640 -1 Falling back to public interface 2024-03-09T13:53:21.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:53:22.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:53:22.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:22.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:53:22.646 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:53:22.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:22.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:53:22.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:53:23.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:53:23.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:23.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:53:23.921 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:53:23.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:23.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:53:23.975 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:23.974+0000 7fe6d681a640 -1 osd.1 62 log_to_monitors true 2024-03-09T13:53:24.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:53:25.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:53:25.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:25.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:53:25.211 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:53:25.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:25.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:53:25.585 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 66 up_thru 66 down_at 63 last_clean_interval [10,62) [v2:127.0.0.1:6810/495490824,v1:127.0.0.1:6811/495490824] [v2:127.0.0.1:6812/495490824,v1:127.0.0.1:6813/495490824] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:53:25.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:53:25.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:53:25.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:53:25.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:53:25.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:53:25.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:53:25.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:53:25.585 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:53:25.585 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:53:25.585 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:53:25.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:53:25.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:53:25.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:53:25.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:53:25.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:53:25.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:53:25.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:53:25.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:53:25.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:53:25.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:53:25.978 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:53:25.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:53:25.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:53:25.978 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:53:26.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836505 2024-03-09T13:53:26.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836505 2024-03-09T13:53:26.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505' 2024-03-09T13:53:26.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:53:26.082 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:53:26.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841538 2024-03-09T13:53:26.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841538 2024-03-09T13:53:26.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836505 1-283467841538' 2024-03-09T13:53:26.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:53:26.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836505 2024-03-09T13:53:26.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:53:26.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:53:26.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836505 2024-03-09T13:53:26.188 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:53:26.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836505 2024-03-09T13:53:26.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-03-09T13:53:26.189 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836505 2024-03-09T13:53:26.189 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:53:26.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836504 -lt 21474836505 2024-03-09T13:53:26.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:53:27.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:53:27.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:53:27.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836506 -lt 21474836505 2024-03-09T13:53:27.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:53:27.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841538 2024-03-09T13:53:27.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:53:27.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:53:27.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841538 2024-03-09T13:53:27.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:53:27.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841538 2024-03-09T13:53:27.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2024-03-09T13:53:27.752 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 283467841538 2024-03-09T13:53:27.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:53:28.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841538 -lt 283467841538 2024-03-09T13:53:28.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:53:28.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:53:28.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:53:28.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:53:28.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:53:28.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:53:28.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:53:28.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:53:28.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:53:28.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:53:28.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:53:28.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:53:28.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:53:28.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:53:28.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:53:29.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:53:29.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:53:29.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:53:29.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:29.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ2 2024-03-09T13:53:29.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 2 % 2 2024-03-09T13:53:29.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T13:53:29.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1045: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1046: TEST_corrupt_scrub_replicated: echo UVWXYZ 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1047: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:53:29.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:53:29.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:53:29.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:53:29.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:53:29.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:53:29.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:53:29.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:53:29.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:53:29.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:53:29.311 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:53:29.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:53:29.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:53:29.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T13:53:31.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:53:31.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:53:31.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:53:31.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:53:31.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:53:31.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:53:31.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:53:31.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:53:31.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:53:31.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:53:31.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:53:31.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:53:31.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:53:31.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:53:31.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:53:31.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:53:31.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:53:31.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:53:31.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:53:31.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:53:31.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:53:31.290 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:53:31.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:53:31.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:53:31.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:53:31.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:53:31.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:53:31.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:53:31.316 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:31.315+0000 7efe1c3e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:53:31.319 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:31.318+0000 7efe1c3e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:53:31.320 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:31.319+0000 7efe1c3e1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:53:31.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:53:31.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:53:31.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:53:31.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:53:31.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:53:31.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:31.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:53:31.572 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:53:31.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:31.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:53:31.836 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:31.835+0000 7efe1c3e1640 -1 Falling back to public interface 2024-03-09T13:53:31.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:53:32.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:53:32.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:32.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:53:32.843 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:53:32.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:32.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:53:33.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:53:34.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:53:34.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:34.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:53:34.118 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:53:34.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:34.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:53:34.166 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:34.166+0000 7efe1c3e1640 -1 osd.0 68 log_to_monitors true 2024-03-09T13:53:34.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:53:35.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:53:35.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:35.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:53:35.398 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:53:35.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:35.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:53:35.669 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [5,68) [v2:127.0.0.1:6802/1233521922,v1:127.0.0.1:6803/1233521922] [v2:127.0.0.1:6804/1233521922,v1:127.0.0.1:6805/1233521922] exists,up f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:53:35.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:53:35.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:53:35.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:53:35.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:53:35.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:53:35.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:53:35.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:53:35.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:53:35.670 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:53:35.670 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:53:35.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:53:35.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:53:35.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:53:35.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:53:35.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:53:35.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:53:35.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:53:35.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:53:35.758 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:53:36.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:53:36.031 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:53:36.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:53:36.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:53:36.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:53:36.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645314 2024-03-09T13:53:36.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645314 2024-03-09T13:53:36.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645314' 2024-03-09T13:53:36.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:53:36.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:53:36.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841541 2024-03-09T13:53:36.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841541 2024-03-09T13:53:36.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645314 1-283467841541' 2024-03-09T13:53:36.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:53:36.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-309237645314 2024-03-09T13:53:36.239 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:53:36.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:53:36.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-309237645314 2024-03-09T13:53:36.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:53:36.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645314 2024-03-09T13:53:36.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 309237645314' 2024-03-09T13:53:36.241 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 309237645314 2024-03-09T13:53:36.242 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:53:36.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 309237645314 2024-03-09T13:53:36.512 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:53:37.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:53:37.514 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:53:37.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645314 -lt 309237645314 2024-03-09T13:53:37.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:53:37.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-283467841541 2024-03-09T13:53:37.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:53:37.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:53:37.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-283467841541 2024-03-09T13:53:37.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:53:37.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841541 2024-03-09T13:53:37.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 283467841541' 2024-03-09T13:53:37.796 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 283467841541 2024-03-09T13:53:37.796 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:53:38.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841541 -lt 283467841541 2024-03-09T13:53:38.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:53:38.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:53:38.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:53:38.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:53:38.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:53:38.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:53:38.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:53:38.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:53:38.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:53:38.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:53:38.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:53:38.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:53:38.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:53:38.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:53:38.680 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:53:39.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:53:39.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:53:39.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:53:39.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:39.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ3 2024-03-09T13:53:39.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 3 % 2 2024-03-09T13:53:39.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T13:53:39.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:53:39.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2024-03-09T13:53:39.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:53:39.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:53:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:53:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:53:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2024-03-09T13:53:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:53:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:53:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:53:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:53:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:53:39.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:53:39.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:53:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:53:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:53:39.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:53:39.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:53:39.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2024-03-09T13:53:39.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:53:39.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:53:39.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:53:39.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:53:39.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:53:39.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2024-03-09T13:53:40.515 INFO:tasks.workunit.client.0.smithi162.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2024-03-09T13:53:41.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:53:41.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:53:41.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:53:41.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:53:41.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:53:41.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:53:41.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:53:41.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:53:41.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:53:41.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:53:41.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:53:41.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:53:41.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:53:41.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:53:41.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:53:41.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:53:41.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:53:41.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:53:41.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:53:41.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:53:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:53:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:53:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:53:41.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:53:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:53:41.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:53:41.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:53:41.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:53:41.066 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:41.066+0000 7f357e20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:53:41.068 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:41.068+0000 7f357e20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:53:41.070 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:41.070+0000 7f357e20f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:53:41.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:53:41.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:53:41.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:53:41.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:53:41.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:53:41.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:41.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:53:41.322 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:53:41.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:41.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:53:41.590 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:41.590+0000 7f357e20f640 -1 Falling back to public interface 2024-03-09T13:53:41.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:53:42.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:53:42.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:42.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:53:42.596 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:53:42.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:42.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:53:42.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:53:43.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:53:43.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:43.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:53:43.873 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:53:43.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:43.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:53:43.922 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:43.922+0000 7f357e20f640 -1 osd.1 73 log_to_monitors true 2024-03-09T13:53:44.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:53:45.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:53:45.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:45.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:53:45.209 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:53:45.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:45.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:53:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [66,73) [v2:127.0.0.1:6810/483331945,v1:127.0.0.1:6811/483331945] [v2:127.0.0.1:6812/483331945,v1:127.0.0.1:6813/483331945] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:53:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:53:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:53:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:53:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:53:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:53:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:53:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:53:45.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:53:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:53:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:53:45.482 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:53:45.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:53:45.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:53:45.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:53:45.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:53:45.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:53:45.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:53:45.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:53:45.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:53:45.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:53:45.844 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:53:45.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:53:45.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:53:45.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:53:45.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645317 2024-03-09T13:53:45.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645317 2024-03-09T13:53:45.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645317' 2024-03-09T13:53:45.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:53:45.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:53:46.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481794 2024-03-09T13:53:46.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481794 2024-03-09T13:53:46.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-309237645317 1-330712481794' 2024-03-09T13:53:46.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:53:46.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-309237645317 2024-03-09T13:53:46.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:53:46.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:53:46.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-309237645317 2024-03-09T13:53:46.055 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:53:46.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645317 2024-03-09T13:53:46.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 309237645317' 2024-03-09T13:53:46.056 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 309237645317 2024-03-09T13:53:46.057 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:53:46.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645316 -lt 309237645317 2024-03-09T13:53:46.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:53:47.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:53:47.329 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:53:47.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645317 -lt 309237645317 2024-03-09T13:53:47.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:53:47.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481794 2024-03-09T13:53:47.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:53:47.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:53:47.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481794 2024-03-09T13:53:47.614 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:53:47.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481794 2024-03-09T13:53:47.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2024-03-09T13:53:47.615 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 330712481794 2024-03-09T13:53:47.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:53:47.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481794 -lt 330712481794 2024-03-09T13:53:47.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:53:47.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:53:47.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:53:48.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:53:48.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:53:48.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:53:48.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:53:48.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:53:48.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:53:48.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:53:48.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:53:48.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:53:48.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:53:48.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:53:48.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:53:48.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:53:48.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:53:48.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:53:48.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:48.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ4 2024-03-09T13:53:48.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 4 % 2 2024-03-09T13:53:48.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T13:53:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:53:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-09T13:53:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:53:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:53:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:53:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:53:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-09T13:53:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:53:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:53:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:53:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:53:48.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:53:48.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:53:48.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:53:48.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:53:48.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:53:48.843 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:53:49.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:53:49.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-09T13:53:49.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:53:49.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:53:49.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:53:49.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:53:49.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:53:49.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2024-03-09T13:53:50.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:53:50.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:53:50.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:53:50.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:53:50.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:53:50.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:53:50.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:53:50.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:53:50.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:53:50.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:53:50.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:53:50.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:53:50.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:53:50.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:53:50.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:53:50.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:53:50.874 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:53:50.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:53:50.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:53:50.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:53:50.877 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:53:50.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:53:50.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:53:50.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:53:50.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:53:50.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:53:50.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:53:50.903 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:50.903+0000 7f7a12173640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:53:50.905 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:50.905+0000 7f7a12173640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:53:50.907 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:50.907+0000 7f7a12173640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:53:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:53:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:53:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:53:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:53:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:53:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:53:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:53:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:51.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:53:51.422 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:51.422+0000 7f7a12173640 -1 Falling back to public interface 2024-03-09T13:53:51.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:53:52.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:53:52.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:52.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:53:52.432 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:53:52.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:52.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:53:52.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:53:53.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:53:53.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:53.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:53:53.761 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:53:53.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:53.761 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:53:53.784 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:53:53.784+0000 7f7a12173640 -1 osd.0 78 log_to_monitors true 2024-03-09T13:53:54.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:53:55.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:53:55.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:53:55.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:53:55.089 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:53:55.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:53:55.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:53:55.371 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [72,78) [v2:127.0.0.1:6802/1288344663,v1:127.0.0.1:6803/1288344663] [v2:127.0.0.1:6804/1288344663,v1:127.0.0.1:6805/1288344663] exists,up f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:53:55.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:53:55.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:53:55.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:53:55.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:53:55.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:53:55.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:53:55.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:53:55.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:53:55.372 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:53:55.372 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:53:55.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:53:55.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:53:55.372 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:53:55.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:53:55.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:53:55.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:53:55.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:53:55.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:53:55.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:53:55.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:53:55.731 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:53:55.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:53:55.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:53:55.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:53:55.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318274 2024-03-09T13:53:55.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318274 2024-03-09T13:53:55.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274' 2024-03-09T13:53:55.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:53:55.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:53:55.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=330712481797 2024-03-09T13:53:55.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 330712481797 2024-03-09T13:53:55.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318274 1-330712481797' 2024-03-09T13:53:55.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:53:55.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318274 2024-03-09T13:53:55.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:53:55.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:53:55.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318274 2024-03-09T13:53:55.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:53:55.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318274 2024-03-09T13:53:55.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318274' 2024-03-09T13:53:55.942 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 352187318274 2024-03-09T13:53:55.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:53:56.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 352187318274 2024-03-09T13:53:56.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:53:57.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:53:57.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:53:57.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318274 -lt 352187318274 2024-03-09T13:53:57.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:53:57.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-330712481797 2024-03-09T13:53:57.500 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:53:57.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:53:57.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-330712481797 2024-03-09T13:53:57.501 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:53:57.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=330712481797 2024-03-09T13:53:57.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 330712481797' 2024-03-09T13:53:57.503 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 330712481797 2024-03-09T13:53:57.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:53:57.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 330712481797 -lt 330712481797 2024-03-09T13:53:57.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:53:57.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:53:57.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:53:58.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:53:58.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:53:58.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:53:58.116 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:53:58.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:53:58.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:53:58.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:53:58.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:53:58.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:53:58.386 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:53:58.386 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:53:58.386 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:53:58.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:53:58.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:53:58.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:53:58.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:53:58.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ5 2024-03-09T13:53:58.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 5 % 2 2024-03-09T13:53:58.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T13:53:58.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:53:58.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-09T13:53:58.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:53:58.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:53:58.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:53:58.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:53:58.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-09T13:53:58.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:53:58.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:53:58.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:53:58.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:53:58.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:53:58.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:53:58.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:53:58.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:53:58.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:53:58.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:53:59.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:53:59.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2024-03-09T13:53:59.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:53:59.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:53:59.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:53:59.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:53:59.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:53:59.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2024-03-09T13:54:00.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:54:00.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:54:00.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:54:00.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:54:00.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:54:00.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:54:00.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:54:00.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:54:00.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:54:00.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:54:00.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:54:00.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:54:00.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:54:00.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:54:00.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:54:00.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:54:00.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:54:00.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:54:00.791 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:54:00.791 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:54:00.791 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:54:00.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:54:00.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:54:00.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:54:00.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:54:00.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:54:00.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:54:00.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:54:00.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:54:00.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:54:00.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:54:00.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:54:00.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:54:00.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:54:00.793 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:54:00.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:54:00.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:54:00.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:54:00.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:54:00.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:54:00.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:54:00.822 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:00.819+0000 7fedaf00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:00.822 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:00.821+0000 7fedaf00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:00.823 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:00.822+0000 7fedaf00f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:01.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:54:01.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:54:01.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:54:01.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:54:01.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:54:01.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:01.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:54:01.075 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:54:01.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:01.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:01.341 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:01.340+0000 7fedaf00f640 -1 Falling back to public interface 2024-03-09T13:54:01.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:02.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:02.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:02.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:54:02.345 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:54:02.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:02.345 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:02.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:03.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:03.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:03.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:54:03.633 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:54:03.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:03.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:03.683 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:03.682+0000 7fedaf00f640 -1 osd.1 84 log_to_monitors true 2024-03-09T13:54:03.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:04.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:04.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:04.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:54:04.977 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:54:04.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:04.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:05.261 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6810/665954341,v1:127.0.0.1:6811/665954341] [v2:127.0.0.1:6812/665954341,v1:127.0.0.1:6813/665954341] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:54:05.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:54:05.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:54:05.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:54:05.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:54:05.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:54:05.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:54:05.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:54:05.261 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:54:05.261 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:54:05.262 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:54:05.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:54:05.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:54:05.262 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:54:05.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:54:05.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:54:05.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:54:05.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:54:05.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:54:05.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:54:05.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:54:05.620 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:54:05.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:54:05.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:54:05.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:54:05.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318277 2024-03-09T13:54:05.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318277 2024-03-09T13:54:05.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277' 2024-03-09T13:54:05.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:54:05.724 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:54:05.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122050 2024-03-09T13:54:05.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122050 2024-03-09T13:54:05.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-352187318277 1-377957122050' 2024-03-09T13:54:05.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:54:05.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-352187318277 2024-03-09T13:54:05.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:54:05.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:54:05.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-352187318277 2024-03-09T13:54:05.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:54:05.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318277 2024-03-09T13:54:05.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 352187318277' 2024-03-09T13:54:05.833 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 352187318277 2024-03-09T13:54:05.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:54:06.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318276 -lt 352187318277 2024-03-09T13:54:06.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:54:07.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:54:07.107 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:54:07.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318277 -lt 352187318277 2024-03-09T13:54:07.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:54:07.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122050 2024-03-09T13:54:07.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:54:07.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:54:07.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122050 2024-03-09T13:54:07.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:54:07.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122050 2024-03-09T13:54:07.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2024-03-09T13:54:07.391 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 377957122050 2024-03-09T13:54:07.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:54:07.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122050 -lt 377957122050 2024-03-09T13:54:07.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:54:07.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:07.667 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:54:08.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:54:08.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:54:08.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:54:08.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:54:08.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:54:08.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:54:08.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:54:08.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:54:08.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:54:08.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:54:08.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:08.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:54:08.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:54:08.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:54:08.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:54:08.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:54:08.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ6 2024-03-09T13:54:08.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 6 % 2 2024-03-09T13:54:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T13:54:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:54:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: echo extra 2024-03-09T13:54:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1068: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-09T13:54:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:54:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:54:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:54:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:54:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-09T13:54:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:54:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:54:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:54:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:54:08.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:54:08.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:54:08.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:54:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:54:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:54:08.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:54:08.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:54:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-09T13:54:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:54:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:54:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:54:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:54:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:54:08.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2024-03-09T13:54:10.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:54:10.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:54:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:54:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:54:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:54:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:54:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:54:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:54:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:54:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:54:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:54:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:54:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:54:10.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:54:10.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:54:10.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:54:10.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:54:10.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:54:10.630 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:54:10.630 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:54:10.630 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:54:10.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:54:10.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:54:10.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:54:10.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:54:10.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:54:10.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:54:10.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:54:10.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:54:10.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:54:10.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:54:10.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:54:10.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:54:10.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:54:10.631 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:54:10.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:54:10.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:54:10.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:54:10.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:54:10.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:54:10.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:54:10.658 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:10.657+0000 7f0ae796d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:10.661 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:10.660+0000 7f0ae796d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:10.662 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:10.661+0000 7f0ae796d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:10.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:54:10.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:54:10.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:54:10.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:54:10.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:54:10.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:10.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:54:10.929 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:54:10.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:10.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:54:11.177 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:11.176+0000 7f0ae796d640 -1 Falling back to public interface 2024-03-09T13:54:11.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:12.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:12.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:12.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:54:12.203 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:54:12.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:12.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:54:12.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:13.478 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:13.477+0000 7f0ae796d640 -1 osd.0 89 log_to_monitors true 2024-03-09T13:54:13.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:13.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:13.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:54:13.487 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:54:13.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:13.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:54:13.793 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:14.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:14.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:14.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:54:14.796 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:54:14.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:14.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:54:15.173 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 93 up_thru 93 down_at 90 last_clean_interval [82,89) [v2:127.0.0.1:6802/3242316001,v1:127.0.0.1:6803/3242316001] [v2:127.0.0.1:6804/3242316001,v1:127.0.0.1:6805/3242316001] exists,up f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:54:15.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:54:15.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:54:15.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:54:15.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:54:15.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:54:15.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:54:15.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:54:15.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:54:15.174 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:54:15.174 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:54:15.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:54:15.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:54:15.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:54:15.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:54:15.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:54:15.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:54:15.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:54:15.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:54:15.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:54:15.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:54:15.534 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:54:15.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:54:15.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:54:15.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:54:15.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958530 2024-03-09T13:54:15.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958530 2024-03-09T13:54:15.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530' 2024-03-09T13:54:15.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:54:15.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:54:15.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=377957122053 2024-03-09T13:54:15.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 377957122053 2024-03-09T13:54:15.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958530 1-377957122053' 2024-03-09T13:54:15.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:54:15.744 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958530 2024-03-09T13:54:15.744 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:54:15.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:54:15.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958530 2024-03-09T13:54:15.745 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:54:15.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958530 2024-03-09T13:54:15.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958530' 2024-03-09T13:54:15.747 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 399431958530 2024-03-09T13:54:15.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:54:16.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 399431958530 2024-03-09T13:54:16.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:54:17.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:54:17.020 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:54:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958530 -lt 399431958530 2024-03-09T13:54:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:54:17.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-377957122053 2024-03-09T13:54:17.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:54:17.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:54:17.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-377957122053 2024-03-09T13:54:17.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:54:17.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=377957122053 2024-03-09T13:54:17.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 377957122053' 2024-03-09T13:54:17.308 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 377957122053 2024-03-09T13:54:17.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:54:17.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 377957122053 -lt 377957122053 2024-03-09T13:54:17.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:54:17.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:17.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:54:17.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:54:17.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:54:17.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:54:17.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:54:17.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:54:17.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:54:17.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:54:17.923 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:54:18.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:54:18.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:54:18.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:18.190 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:54:18.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:54:18.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:54:18.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:54:18.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2024-03-09T13:54:18.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:54:18.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ7 2024-03-09T13:54:18.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 7 % 2 2024-03-09T13:54:18.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T13:54:18.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:54:18.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: echo -n newheader 2024-03-09T13:54:18.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-09T13:54:18.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:54:18.529 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:54:18.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:54:18.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:54:18.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-09T13:54:18.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:54:18.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:54:18.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:54:18.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:54:18.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:54:18.530 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:54:18.530 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:54:18.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:54:18.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:54:18.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:54:18.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:54:18.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-09T13:54:18.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:54:18.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:54:18.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:54:18.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:54:18.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:54:18.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2024-03-09T13:54:20.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:54:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:54:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:54:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:54:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:54:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:54:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:54:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:54:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:54:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:54:20.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:54:20.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:54:20.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:54:20.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:54:20.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:54:20.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:54:20.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:54:20.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:54:20.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:54:20.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:54:20.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:54:20.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:54:20.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:54:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:54:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:54:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:54:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:54:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:54:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:54:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:54:20.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:54:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:54:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:54:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:54:20.451 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:54:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:54:20.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:54:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:54:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:54:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:54:20.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:54:20.477 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:20.476+0000 7efe7bace640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:20.479 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:20.478+0000 7efe7bace640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:20.481 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:20.480+0000 7efe7bace640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:20.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:54:20.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:54:20.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:54:20.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:54:20.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:54:20.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:20.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:54:20.746 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:54:20.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:20.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:20.996 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:20.995+0000 7efe7bace640 -1 Falling back to public interface 2024-03-09T13:54:21.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:22.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:54:22.017 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:54:22.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:22.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:22.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:23.287 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:23.286+0000 7efe7bace640 -1 osd.1 95 log_to_monitors true 2024-03-09T13:54:23.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:23.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:23.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:54:23.303 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:54:23.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:23.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:23.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:24.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:24.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:24.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:54:24.666 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:54:24.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:24.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:24.965 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 99 up_thru 99 down_at 96 last_clean_interval [88,95) [v2:127.0.0.1:6810/371140851,v1:127.0.0.1:6811/371140851] [v2:127.0.0.1:6812/371140851,v1:127.0.0.1:6813/371140851] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:54:24.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:54:24.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:54:24.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:54:24.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:54:24.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:54:24.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:54:24.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:54:24.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:54:24.966 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:54:24.966 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:54:24.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:54:24.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:54:24.966 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:54:25.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:54:25.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:54:25.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:54:25.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:54:25.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:54:25.054 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:54:25.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:54:25.323 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:54:25.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:54:25.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:54:25.324 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:54:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=399431958533 2024-03-09T13:54:25.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 399431958533 2024-03-09T13:54:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533' 2024-03-09T13:54:25.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:54:25.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:54:25.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762306 2024-03-09T13:54:25.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762306 2024-03-09T13:54:25.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-399431958533 1-425201762306' 2024-03-09T13:54:25.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:54:25.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-399431958533 2024-03-09T13:54:25.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:54:25.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:54:25.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-399431958533 2024-03-09T13:54:25.536 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:54:25.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=399431958533 2024-03-09T13:54:25.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 399431958533' 2024-03-09T13:54:25.537 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 399431958533 2024-03-09T13:54:25.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:54:25.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958532 -lt 399431958533 2024-03-09T13:54:25.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:54:26.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:54:26.811 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:54:27.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 399431958533 -lt 399431958533 2024-03-09T13:54:27.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:54:27.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762306 2024-03-09T13:54:27.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:54:27.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:54:27.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762306 2024-03-09T13:54:27.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:54:27.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762306 2024-03-09T13:54:27.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762306' 2024-03-09T13:54:27.095 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 425201762306 2024-03-09T13:54:27.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:54:27.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762306 -lt 425201762306 2024-03-09T13:54:27.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:54:27.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:27.369 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:54:27.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:54:27.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:54:27.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:54:27.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:54:27.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:54:27.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:54:27.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:54:27.711 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:54:27.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:54:27.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:54:27.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:27.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:54:28.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:54:28.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:54:28.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:54:28.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2024-03-09T13:54:28.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:54:28.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ8 2024-03-09T13:54:28.322 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 8 % 2 2024-03-09T13:54:28.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T13:54:28.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:54:28.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1080: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2024-03-09T13:54:28.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1081: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2024-03-09T13:54:28.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-09T13:54:28.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1085: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-09T13:54:28.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:54:28.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:54:28.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:54:28.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:54:28.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-09T13:54:28.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:54:28.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:54:28.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:54:28.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:54:28.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:54:28.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:54:28.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:54:28.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:54:28.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:54:28.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:54:28.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:54:28.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-09T13:54:28.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:54:28.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:54:28.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:54:28.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:54:28.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:54:28.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2024-03-09T13:54:30.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:54:30.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:54:30.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:54:30.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:54:30.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:54:30.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:54:30.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:54:30.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:54:30.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:54:30.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:54:30.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:54:30.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:54:30.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:54:30.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:54:30.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:54:30.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:54:30.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:54:30.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:54:30.483 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:54:30.483 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:54:30.483 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:54:30.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:54:30.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:54:30.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:54:30.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:54:30.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:54:30.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:54:30.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:54:30.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:54:30.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:54:30.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:54:30.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:54:30.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:54:30.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:54:30.485 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:54:30.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:54:30.485 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:54:30.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:54:30.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:54:30.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:54:30.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:54:30.510 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:30.510+0000 7efd65ffd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:30.512 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:30.512+0000 7efd65ffd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:30.513 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:30.513+0000 7efd65ffd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:30.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:54:30.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:54:30.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:54:30.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:54:30.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:54:30.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:30.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:54:30.770 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:54:30.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:30.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:54:31.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:31.035+0000 7efd65ffd640 -1 Falling back to public interface 2024-03-09T13:54:31.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:32.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:32.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:32.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:54:32.081 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:54:32.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:32.082 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:54:32.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:33.322 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:33.322+0000 7efd65ffd640 -1 osd.0 101 log_to_monitors true 2024-03-09T13:54:33.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:33.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:33.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:54:33.362 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:54:33.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:33.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:54:33.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:34.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:34.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:34.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:54:34.713 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:54:34.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:34.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:54:35.055 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 105 up_thru 105 down_at 102 last_clean_interval [93,101) [v2:127.0.0.1:6802/1027043477,v1:127.0.0.1:6803/1027043477] [v2:127.0.0.1:6804/1027043477,v1:127.0.0.1:6805/1027043477] exists,up f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:54:35.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:54:35.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:54:35.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:54:35.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:54:35.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:54:35.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:54:35.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:54:35.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:54:35.056 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:54:35.056 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:54:35.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:54:35.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:54:35.056 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:54:35.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:54:35.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:54:35.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:54:35.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:54:35.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:54:35.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:54:35.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:54:35.418 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:54:35.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:54:35.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:54:35.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:54:35.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566082 2024-03-09T13:54:35.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566082 2024-03-09T13:54:35.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082' 2024-03-09T13:54:35.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:54:35.523 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:54:35.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=425201762309 2024-03-09T13:54:35.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 425201762309 2024-03-09T13:54:35.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566082 1-425201762309' 2024-03-09T13:54:35.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:54:35.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566082 2024-03-09T13:54:35.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:54:35.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:54:35.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566082 2024-03-09T13:54:35.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:54:35.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566082 2024-03-09T13:54:35.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566082' 2024-03-09T13:54:35.629 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 450971566082 2024-03-09T13:54:35.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:54:35.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 450971566082 2024-03-09T13:54:35.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:54:36.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:54:36.902 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:54:37.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566082 -lt 450971566082 2024-03-09T13:54:37.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:54:37.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-425201762309 2024-03-09T13:54:37.184 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:54:37.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:54:37.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-425201762309 2024-03-09T13:54:37.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:54:37.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=425201762309 2024-03-09T13:54:37.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 425201762309' 2024-03-09T13:54:37.187 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 425201762309 2024-03-09T13:54:37.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:54:37.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 425201762309 -lt 425201762309 2024-03-09T13:54:37.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:54:37.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:37.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:54:37.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:54:37.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:54:37.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:54:37.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:54:37.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:54:37.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:54:37.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:54:37.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:54:38.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:54:38.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:54:38.071 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:38.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:54:38.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:54:38.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:54:38.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:54:38.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-09T13:54:38.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:54:38.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:54:38.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:54:38.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:54:38.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-09T13:54:38.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:54:38.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:54:38.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:54:38.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:54:38.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:54:38.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:54:38.412 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:54:38.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:54:38.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:54:38.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:54:38.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:54:38.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-09T13:54:38.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:54:38.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:54:38.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:54:38.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:54:38.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:54:38.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2024-03-09T13:54:40.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:54:40.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:54:40.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:54:40.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:54:40.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:54:40.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:54:40.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:54:40.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:54:40.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:54:40.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:54:40.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:54:40.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:54:40.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:54:40.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:54:40.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:54:40.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:54:40.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:54:40.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:54:40.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:54:40.325 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:54:40.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:54:40.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:54:40.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:54:40.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:54:40.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:54:40.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:54:40.352 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:40.352+0000 7f43d4b8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:40.354 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:40.354+0000 7f43d4b8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:40.355 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:40.355+0000 7f43d4b8c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:40.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:54:40.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:54:40.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:54:40.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:54:40.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:54:40.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:40.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:54:40.625 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:54:40.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:40.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:40.881 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:40.881+0000 7f43d4b8c640 -1 Falling back to public interface 2024-03-09T13:54:40.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:41.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:41.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:41.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:54:41.935 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:54:41.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:41.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:42.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:54:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:54:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:43.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:43.226 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:43.227+0000 7f43d4b8c640 -1 osd.1 107 log_to_monitors true 2024-03-09T13:54:43.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:54:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:54:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:44.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:45.006 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 111 up_thru 111 down_at 108 last_clean_interval [99,107) [v2:127.0.0.1:6810/73970839,v1:127.0.0.1:6811/73970839] [v2:127.0.0.1:6812/73970839,v1:127.0.0.1:6813/73970839] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:54:45.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:54:45.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:54:45.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:54:45.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:54:45.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:54:45.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:54:45.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:54:45.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:54:45.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:54:45.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:54:45.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:54:45.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:54:45.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:54:45.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:54:45.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:54:45.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:54:45.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:54:45.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:54:45.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:54:45.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:54:45.371 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:54:45.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:54:45.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:54:45.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:54:45.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566085 2024-03-09T13:54:45.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566085 2024-03-09T13:54:45.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085' 2024-03-09T13:54:45.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:54:45.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:54:45.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=476741369858 2024-03-09T13:54:45.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 476741369858 2024-03-09T13:54:45.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566085 1-476741369858' 2024-03-09T13:54:45.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:54:45.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566085 2024-03-09T13:54:45.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:54:45.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:54:45.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566085 2024-03-09T13:54:45.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:54:45.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566085 2024-03-09T13:54:45.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566085' 2024-03-09T13:54:45.582 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 450971566085 2024-03-09T13:54:45.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:54:45.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566084 -lt 450971566085 2024-03-09T13:54:45.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:54:46.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:54:46.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:54:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566085 -lt 450971566085 2024-03-09T13:54:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:54:47.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-476741369858 2024-03-09T13:54:47.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:54:47.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:54:47.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-476741369858 2024-03-09T13:54:47.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:54:47.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=476741369858 2024-03-09T13:54:47.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 476741369858' 2024-03-09T13:54:47.135 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 476741369858 2024-03-09T13:54:47.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:54:47.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 476741369858 -lt 476741369858 2024-03-09T13:54:47.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:54:47.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:47.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:54:47.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:54:47.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:54:47.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:54:47.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:54:47.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:54:47.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:54:47.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:54:47.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:54:48.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:54:48.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:54:48.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:48.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:54:48.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:54:48.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:54:48.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1087: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:54:48.370 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:54:48.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:54:48.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:54:48.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:54:48.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:54:48.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-09T13:54:48.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:54:48.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:54:48.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:54:48.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:54:48.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:54:48.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2024-03-09T13:54:50.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:54:50.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:54:50.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:54:50.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:54:50.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:54:50.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:54:50.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:54:50.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:54:50.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:54:50.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:54:50.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:54:50.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:54:50.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:54:50.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:54:50.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:54:50.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:54:50.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:54:50.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:54:50.210 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:54:50.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:54:50.211 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:54:50.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:54:50.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:54:50.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:54:50.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:54:50.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:54:50.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:54:50.238 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:50.237+0000 7f741c213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:50.240 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:50.239+0000 7f741c213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:50.241 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:50.241+0000 7f741c213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:54:50.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:54:50.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:54:50.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:54:50.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:54:50.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:54:50.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:50.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:54:50.498 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:54:50.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:50.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:50.760 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:50.759+0000 7f741c213640 -1 Falling back to public interface 2024-03-09T13:54:50.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:51.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:51.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:51.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:54:51.802 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:54:51.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:51.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:52.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:53.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:53.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:53.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:54:53.086 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:54:53.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:53.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:53.317 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:54:53.316+0000 7f741c213640 -1 osd.1 113 log_to_monitors true 2024-03-09T13:54:53.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:54:54.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:54:54.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:54:54.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:54:54.373 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:54:54.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:54:54.373 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:54:54.665 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 117 up_thru 117 down_at 114 last_clean_interval [111,113) [v2:127.0.0.1:6810/2259271933,v1:127.0.0.1:6811/2259271933] [v2:127.0.0.1:6812/2259271933,v1:127.0.0.1:6813/2259271933] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:54:54.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:54:54.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:54:54.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:54:54.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:54:54.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:54:54.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:54:54.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:54:54.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:54:54.665 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:54:54.665 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:54:54.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:54:54.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:54:54.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:54:54.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:54:54.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:54:54.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:54:54.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:54:54.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:54:54.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:54:55.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:54:55.031 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:54:55.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:54:55.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:54:55.032 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:54:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566088 2024-03-09T13:54:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566088 2024-03-09T13:54:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566088' 2024-03-09T13:54:55.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:54:55.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:54:55.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=502511173634 2024-03-09T13:54:55.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 502511173634 2024-03-09T13:54:55.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566088 1-502511173634' 2024-03-09T13:54:55.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:54:55.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566088 2024-03-09T13:54:55.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:54:55.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:54:55.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566088 2024-03-09T13:54:55.283 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:54:55.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566088 2024-03-09T13:54:55.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566088' 2024-03-09T13:54:55.284 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 450971566088 2024-03-09T13:54:55.284 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:54:55.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566087 -lt 450971566088 2024-03-09T13:54:55.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:54:56.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:54:56.559 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:54:56.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566088 -lt 450971566088 2024-03-09T13:54:56.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:54:56.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-502511173634 2024-03-09T13:54:56.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:54:56.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:54:56.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-502511173634 2024-03-09T13:54:56.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:54:56.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=502511173634 2024-03-09T13:54:56.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 502511173634' 2024-03-09T13:54:56.873 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 502511173634 2024-03-09T13:54:56.873 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:54:57.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 502511173634 -lt 502511173634 2024-03-09T13:54:57.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:54:57.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:57.146 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:54:57.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:54:57.485 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:54:57.485 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:54:57.485 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:54:57.486 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:54:57.486 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:54:57.486 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:54:57.486 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:54:57.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:54:57.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:54:57.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:57.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:54:58.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T13:54:58.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T13:54:58.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:54:58.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T13:54:58.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:54:58.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:54:58.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:54:58.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:54:58.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:54:58.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:54:58.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:54:58.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:54:58.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:54:58.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:54:58.506 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:54:58.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:54:58.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:58.507 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:54:58.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T13:54:58.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:54:58.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:54:58.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:54:58.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:54:58.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:54:58.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:54:58.849 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:54:58.849 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:54:59.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:54:59.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:54:59.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:54:59.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:54:59.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:54:59.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:54:59.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:54:59.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:54:59.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:54:59.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:54:59.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:54:59.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:54:59.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:54:59.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:55:00.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:55:00.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:55:00.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:55:00.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2024-03-09T13:55:00.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:55:00.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-09T13:55:00.004 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 9 % 2 2024-03-09T13:55:00.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T13:55:00.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:55:00.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1093: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-09T13:55:00.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:55:00.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:55:00.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:55:00.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:55:00.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-09T13:55:00.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:55:00.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:55:00.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:55:00.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:55:00.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:55:00.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:55:00.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:55:00.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:55:00.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:55:00.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:55:00.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:55:00.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2024-03-09T13:55:00.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:55:00.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:55:00.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:55:00.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:55:00.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:55:00.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2024-03-09T13:55:02.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:55:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:55:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:55:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:55:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:55:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:55:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:55:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:55:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:55:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:55:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:55:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:55:02.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:55:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:55:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:55:02.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:55:02.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:55:02.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:55:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:55:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:55:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:55:02.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:55:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:55:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:55:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:55:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:55:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:55:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:55:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:55:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:55:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:55:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:55:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:55:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:55:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:55:02.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:55:02.095 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:55:02.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:55:02.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:55:02.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:55:02.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:55:02.121 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:02.120+0000 7f565f81f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:02.123 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:02.123+0000 7f565f81f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:02.125 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:02.124+0000 7f565f81f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:02.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:55:02.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:55:02.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:55:02.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:55:02.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:55:02.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:02.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:55:02.379 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:55:02.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:02.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:02.641 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:02.640+0000 7f565f81f640 -1 Falling back to public interface 2024-03-09T13:55:02.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:03.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:03.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:03.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:55:03.705 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:55:03.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:03.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:03.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:04.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:04.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:04.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:55:04.983 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:55:04.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:04.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:05.048 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:05.047+0000 7f565f81f640 -1 osd.1 119 log_to_monitors true 2024-03-09T13:55:05.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:06.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:06.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:06.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:55:06.352 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:55:06.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:06.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:06.640 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 123 up_thru 123 down_at 120 last_clean_interval [117,119) [v2:127.0.0.1:6810/2304016213,v1:127.0.0.1:6811/2304016213] [v2:127.0.0.1:6812/2304016213,v1:127.0.0.1:6813/2304016213] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:55:06.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:55:06.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:55:06.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:55:06.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:55:06.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:55:06.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:55:06.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:55:06.640 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:55:06.641 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:55:06.641 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:55:06.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:55:06.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:55:06.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:55:06.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:55:06.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:55:06.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:55:06.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:55:06.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:55:06.729 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:55:07.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:55:07.019 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:55:07.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:55:07.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:55:07.019 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:55:07.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566091 2024-03-09T13:55:07.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566091 2024-03-09T13:55:07.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566091' 2024-03-09T13:55:07.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:55:07.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:55:07.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=528280977410 2024-03-09T13:55:07.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 528280977410 2024-03-09T13:55:07.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566091 1-528280977410' 2024-03-09T13:55:07.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:55:07.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566091 2024-03-09T13:55:07.227 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:55:07.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:55:07.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566091 2024-03-09T13:55:07.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:55:07.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566091 2024-03-09T13:55:07.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566091' 2024-03-09T13:55:07.230 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 450971566091 2024-03-09T13:55:07.230 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:55:07.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566090 -lt 450971566091 2024-03-09T13:55:07.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:55:08.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:55:08.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:55:08.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566092 -lt 450971566091 2024-03-09T13:55:08.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:55:08.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-528280977410 2024-03-09T13:55:08.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:55:08.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:55:08.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-528280977410 2024-03-09T13:55:08.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:55:08.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=528280977410 2024-03-09T13:55:08.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 528280977410' 2024-03-09T13:55:08.794 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 528280977410 2024-03-09T13:55:08.794 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:55:09.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 528280977410 -lt 528280977410 2024-03-09T13:55:09.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:55:09.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:55:09.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:55:09.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:55:09.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:55:09.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:55:09.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:55:09.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:55:09.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:55:09.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:55:09.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:55:09.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:55:09.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:55:09.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:55:09.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:55:10.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:55:10.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:55:10.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:55:10.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: echo -n D 2024-03-09T13:55:10.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1095: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-03-09T13:55:10.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1096: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T13:55:10.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:55:10.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:55:10.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:55:10.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:55:10.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T13:55:10.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:55:10.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:55:10.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:55:10.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:55:10.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:55:10.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:55:10.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:55:10.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:55:10.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:55:10.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:55:10.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:55:10.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T13:55:10.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:55:10.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:55:10.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:55:10.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:55:10.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:55:10.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T13:55:12.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:55:12.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:55:12.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:55:12.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:55:12.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:55:12.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:55:12.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:55:12.133 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:55:12.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:55:12.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:55:12.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:55:12.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:55:12.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:55:12.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:55:12.159 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:12.159+0000 7f57635ff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:12.161 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:12.161+0000 7f57635ff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:12.162 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:12.162+0000 7f57635ff640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:12.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:55:12.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:55:12.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:55:12.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:55:12.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:55:12.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:12.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:55:12.414 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:55:12.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:12.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:12.677 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:12.677+0000 7f57635ff640 -1 Falling back to public interface 2024-03-09T13:55:12.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:13.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:13.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:13.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:55:13.698 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:55:13.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:13.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:13.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:14.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:14.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:14.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:55:14.980 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:55:14.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:14.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:15.100 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:15.100+0000 7f57635ff640 -1 osd.1 125 log_to_monitors true 2024-03-09T13:55:15.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:16.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:16.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:16.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:55:16.287 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:55:16.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:16.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:16.566 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 129 up_thru 129 down_at 126 last_clean_interval [123,125) [v2:127.0.0.1:6810/794586224,v1:127.0.0.1:6811/794586224] [v2:127.0.0.1:6812/794586224,v1:127.0.0.1:6813/794586224] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:55:16.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:55:16.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:55:16.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:55:16.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:55:16.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:55:16.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:55:16.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:55:16.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:55:16.567 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:55:16.567 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:55:16.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:55:16.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:55:16.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:55:16.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:55:16.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:55:16.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:55:16.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:55:16.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:55:16.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:55:16.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:55:16.927 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:55:16.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:55:16.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:55:16.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:55:17.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566094 2024-03-09T13:55:17.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566094 2024-03-09T13:55:17.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566094' 2024-03-09T13:55:17.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:55:17.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:55:17.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=554050781186 2024-03-09T13:55:17.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 554050781186 2024-03-09T13:55:17.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-450971566094 1-554050781186' 2024-03-09T13:55:17.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:55:17.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-450971566094 2024-03-09T13:55:17.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:55:17.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:55:17.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-450971566094 2024-03-09T13:55:17.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:55:17.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566094 2024-03-09T13:55:17.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 450971566094' 2024-03-09T13:55:17.138 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 450971566094 2024-03-09T13:55:17.139 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:55:17.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566093 -lt 450971566094 2024-03-09T13:55:17.411 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:55:18.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:55:18.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:55:18.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566095 -lt 450971566094 2024-03-09T13:55:18.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:55:18.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-554050781186 2024-03-09T13:55:18.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:55:18.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:55:18.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-554050781186 2024-03-09T13:55:18.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:55:18.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=554050781186 2024-03-09T13:55:18.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 554050781186' 2024-03-09T13:55:18.699 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 554050781186 2024-03-09T13:55:18.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:55:18.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 554050781186 -lt 554050781186 2024-03-09T13:55:18.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:55:18.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:55:18.975 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:55:19.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:55:19.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:55:19.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:55:19.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:55:19.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:55:19.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:55:19.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:55:19.313 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:55:19.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:55:19.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:55:19.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:55:19.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:55:19.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:55:19.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:55:19.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:55:19.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-03-09T13:55:19.926 INFO:tasks.workunit.client.0.smithi162.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-03-09T13:55:19.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:55:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-09T13:55:19.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 10 % 2 2024-03-09T13:55:19.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T13:55:19.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:55:19.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:55:19.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ11 2024-03-09T13:55:19.928 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 11 % 2 2024-03-09T13:55:19.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T13:55:19.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:55:19.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:55:19.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ12 2024-03-09T13:55:19.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 12 % 2 2024-03-09T13:55:19.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T13:55:19.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:55:19.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:55:19.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ13 2024-03-09T13:55:19.931 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 13 % 2 2024-03-09T13:55:19.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T13:55:19.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:55:19.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:55:19.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ14 2024-03-09T13:55:19.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 14 % 2 2024-03-09T13:55:19.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T13:55:19.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:55:19.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-09T13:55:19.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1107: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-09T13:55:19.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:55:19.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:55:19.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:55:19.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:55:19.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-09T13:55:19.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:55:19.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:55:19.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:55:19.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:55:19.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:55:19.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:55:19.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:55:19.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:55:19.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:55:19.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:55:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:55:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-09T13:55:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:55:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:55:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:55:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:55:20.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:55:20.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2024-03-09T13:55:22.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:55:22.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:55:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:55:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:55:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:55:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:55:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:55:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:55:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:55:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:55:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:55:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:55:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:55:22.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:55:22.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:55:22.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:55:22.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:55:22.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:55:22.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:55:22.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:55:22.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:55:22.017 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:55:22.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:55:22.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:55:22.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:55:22.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:55:22.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:55:22.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:55:22.044 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:22.043+0000 7f5efd993640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:22.046 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:22.046+0000 7f5efd993640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:22.047 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:22.047+0000 7f5efd993640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:22.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:55:22.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:55:22.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:55:22.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:55:22.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:55:22.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:22.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:55:22.297 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:55:22.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:22.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:55:22.562 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:22.562+0000 7f5efd993640 -1 Falling back to public interface 2024-03-09T13:55:22.570 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:23.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:23.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:23.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:55:23.572 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:55:23.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:23.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:55:23.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:24.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:24.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:24.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:55:24.849 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:55:24.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:24.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:55:25.058 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:25.058+0000 7f5efd993640 -1 osd.0 131 log_to_monitors true 2024-03-09T13:55:25.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:26.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:26.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:26.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:55:26.171 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:55:26.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:26.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:55:26.451 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 135 up_thru 135 down_at 132 last_clean_interval [105,131) [v2:127.0.0.1:6802/2166495041,v1:127.0.0.1:6803/2166495041] [v2:127.0.0.1:6804/2166495041,v1:127.0.0.1:6805/2166495041] exists,up f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:55:26.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:55:26.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:55:26.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:55:26.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:55:26.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:55:26.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:55:26.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:55:26.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:55:26.452 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:55:26.452 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:55:26.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:55:26.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:55:26.452 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:55:26.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:55:26.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:55:26.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:55:26.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:55:26.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:55:26.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:55:26.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:55:26.912 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:55:26.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:55:26.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:55:26.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:55:27.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=579820584962 2024-03-09T13:55:27.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 579820584962 2024-03-09T13:55:27.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584962' 2024-03-09T13:55:27.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:55:27.044 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:55:27.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=554050781189 2024-03-09T13:55:27.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 554050781189 2024-03-09T13:55:27.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584962 1-554050781189' 2024-03-09T13:55:27.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:55:27.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-579820584962 2024-03-09T13:55:27.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:55:27.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:55:27.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-579820584962 2024-03-09T13:55:27.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:55:27.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=579820584962 2024-03-09T13:55:27.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 579820584962' 2024-03-09T13:55:27.151 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 579820584962 2024-03-09T13:55:27.152 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:55:27.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 579820584962 2024-03-09T13:55:27.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:55:28.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:55:28.423 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:55:28.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584962 -lt 579820584962 2024-03-09T13:55:28.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:55:28.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-554050781189 2024-03-09T13:55:28.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:55:28.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:55:28.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-554050781189 2024-03-09T13:55:28.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:55:28.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=554050781189 2024-03-09T13:55:28.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 554050781189' 2024-03-09T13:55:28.710 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 554050781189 2024-03-09T13:55:28.710 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:55:28.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 554050781189 -lt 554050781189 2024-03-09T13:55:28.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:55:28.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:55:28.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:55:29.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:55:29.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:55:29.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:55:29.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:55:29.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:55:29.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:55:29.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:55:29.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:55:29.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:55:29.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:55:29.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:55:29.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:55:29.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:55:29.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:55:29.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:55:29.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1108: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-09T13:55:29.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:55:29.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:55:30.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:55:30.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2024-03-09T13:55:30.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:55:30.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:55:30.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:55:30.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:55:30.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:55:30.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2024-03-09T13:55:32.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:55:32.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:55:32.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:55:32.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:55:32.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:55:32.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:55:32.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:55:32.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:55:32.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:55:32.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:55:32.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:55:32.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:55:32.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:55:32.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:55:32.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:55:32.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:55:32.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:55:32.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:55:32.062 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:55:32.062 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:55:32.062 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:55:32.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:55:32.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:55:32.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:55:32.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:55:32.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:55:32.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:55:32.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:55:32.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:55:32.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:55:32.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:55:32.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:55:32.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:55:32.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:55:32.064 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:55:32.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:55:32.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:55:32.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:55:32.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:55:32.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:55:32.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:55:32.090 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:32.090+0000 7f10dfaa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:32.092 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:32.092+0000 7f10dfaa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:32.093 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:32.093+0000 7f10dfaa2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:32.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:55:32.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:55:32.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:55:32.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:55:32.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:55:32.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:32.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:55:32.347 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:55:32.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:32.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:32.614 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:32.614+0000 7f10dfaa2640 -1 Falling back to public interface 2024-03-09T13:55:32.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:33.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:55:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:55:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:33.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:33.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:34.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:34.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:34.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:55:34.923 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:55:34.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:34.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:35.015 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:35.014+0000 7f10dfaa2640 -1 osd.1 136 log_to_monitors true 2024-03-09T13:55:35.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:36.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:36.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:36.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:55:36.304 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:55:36.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:36.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:36.601 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 140 up_thru 140 down_at 137 last_clean_interval [129,136) [v2:127.0.0.1:6810/1863413502,v1:127.0.0.1:6811/1863413502] [v2:127.0.0.1:6812/1863413502,v1:127.0.0.1:6813/1863413502] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:55:36.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:55:36.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:55:36.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:55:36.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:55:36.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:55:36.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:55:36.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:55:36.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:55:36.602 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:55:36.602 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:55:36.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:55:36.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:55:36.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:55:36.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:55:36.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:55:36.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:55:36.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:55:36.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:55:36.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:55:36.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:55:36.961 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:55:36.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:55:36.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:55:36.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:55:37.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=579820584965 2024-03-09T13:55:37.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 579820584965 2024-03-09T13:55:37.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584965' 2024-03-09T13:55:37.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:55:37.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:55:37.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=601295421442 2024-03-09T13:55:37.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 601295421442 2024-03-09T13:55:37.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584965 1-601295421442' 2024-03-09T13:55:37.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:55:37.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-579820584965 2024-03-09T13:55:37.181 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:55:37.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:55:37.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-579820584965 2024-03-09T13:55:37.183 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:55:37.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=579820584965 2024-03-09T13:55:37.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 579820584965' 2024-03-09T13:55:37.185 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 579820584965 2024-03-09T13:55:37.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:55:37.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584964 -lt 579820584965 2024-03-09T13:55:37.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:55:38.455 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:55:38.455 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:55:38.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584965 -lt 579820584965 2024-03-09T13:55:38.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:55:38.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-601295421442 2024-03-09T13:55:38.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:55:38.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:55:38.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-601295421442 2024-03-09T13:55:38.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:55:38.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=601295421442 2024-03-09T13:55:38.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 601295421442' 2024-03-09T13:55:38.758 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 601295421442 2024-03-09T13:55:38.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:55:39.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 601295421442 -lt 601295421442 2024-03-09T13:55:39.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:55:39.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:55:39.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:55:39.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:55:39.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:55:39.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:55:39.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:55:39.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:55:39.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:55:39.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:55:39.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:55:39.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:55:39.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:55:39.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:55:39.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:55:39.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:55:39.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:55:39.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:55:39.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1109: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2024-03-09T13:55:39.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:55:39.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ15 2024-03-09T13:55:39.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 15 % 2 2024-03-09T13:55:39.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T13:55:39.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:55:39.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1113: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-09T13:55:39.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:55:39.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:55:39.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:55:39.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:55:39.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-09T13:55:39.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:55:39.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:55:39.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:55:39.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:55:39.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:55:39.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:55:39.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:55:39.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:55:39.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:55:39.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:55:40.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:55:40.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2024-03-09T13:55:40.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:55:40.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:55:40.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:55:40.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:55:40.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:55:40.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2024-03-09T13:55:41.435 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T13:55:41.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:55:41.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:55:41.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:55:41.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:55:41.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:55:41.954 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:55:41.954 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:55:41.954 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:55:41.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:55:41.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:55:41.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:55:41.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:55:41.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:55:41.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:55:41.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:55:41.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:55:41.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:55:41.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:55:41.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:55:41.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:55:41.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:55:41.955 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:55:41.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:55:41.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:55:41.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:55:41.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:55:41.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:55:41.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:55:41.982 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:41.980+0000 7fbde5519640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:41.984 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:41.983+0000 7fbde5519640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:41.985 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:41.984+0000 7fbde5519640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:42.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:55:42.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:55:42.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:55:42.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:55:42.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:55:42.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:42.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:55:42.240 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:55:42.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:42.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:42.500 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:42.499+0000 7fbde5519640 -1 Falling back to public interface 2024-03-09T13:55:42.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:43.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:43.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:43.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:55:43.516 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:55:43.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:43.516 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:43.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:44.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:44.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:44.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:55:44.794 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:55:44.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:44.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:44.919 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:44.918+0000 7fbde5519640 -1 osd.1 142 log_to_monitors true 2024-03-09T13:55:45.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:46.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:46.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:46.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:55:46.104 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:55:46.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:46.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:55:46.444 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 146 up_thru 146 down_at 143 last_clean_interval [140,142) [v2:127.0.0.1:6810/1690201378,v1:127.0.0.1:6811/1690201378] [v2:127.0.0.1:6812/1690201378,v1:127.0.0.1:6813/1690201378] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:55:46.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:55:46.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:55:46.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:55:46.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:55:46.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:55:46.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:55:46.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:55:46.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:55:46.446 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:55:46.446 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:55:46.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:55:46.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:55:46.446 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:55:46.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:55:46.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:55:46.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:55:46.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:55:46.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:55:46.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:55:46.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:55:46.818 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:55:46.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:55:46.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:55:46.819 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:55:46.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=579820584968 2024-03-09T13:55:46.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 579820584968 2024-03-09T13:55:46.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584968' 2024-03-09T13:55:46.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:55:46.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:55:47.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=627065225218 2024-03-09T13:55:47.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 627065225218 2024-03-09T13:55:47.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-579820584968 1-627065225218' 2024-03-09T13:55:47.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:55:47.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-579820584968 2024-03-09T13:55:47.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:55:47.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:55:47.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-579820584968 2024-03-09T13:55:47.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:55:47.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=579820584968 2024-03-09T13:55:47.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 579820584968' 2024-03-09T13:55:47.065 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 579820584968 2024-03-09T13:55:47.066 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:55:47.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584967 -lt 579820584968 2024-03-09T13:55:47.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:55:48.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:55:48.340 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:55:48.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584967 -lt 579820584968 2024-03-09T13:55:48.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:55:49.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:55:49.646 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:55:49.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 579820584968 -lt 579820584968 2024-03-09T13:55:49.926 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:55:49.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-627065225218 2024-03-09T13:55:49.927 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:55:49.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:55:49.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-627065225218 2024-03-09T13:55:49.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:55:49.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=627065225218 2024-03-09T13:55:49.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 627065225218' 2024-03-09T13:55:49.930 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 627065225218 2024-03-09T13:55:49.930 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:55:50.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 627065225218 -lt 627065225218 2024-03-09T13:55:50.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:55:50.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:55:50.206 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:55:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:55:50.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:55:50.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:55:50.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:55:50.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:55:50.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:55:50.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:55:50.544 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:55:50.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:55:50.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:55:50.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:55:50.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:55:51.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:55:51.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:55:51.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:55:51.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:55:51.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ16 2024-03-09T13:55:51.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 16 % 2 2024-03-09T13:55:51.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T13:55:51.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:55:51.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1117: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-09T13:55:51.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:55:51.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:55:51.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:55:51.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:55:51.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-09T13:55:51.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:55:51.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:55:51.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:55:51.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:55:51.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:55:51.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:55:51.167 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:55:51.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:55:51.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:55:51.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:55:51.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:55:51.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2024-03-09T13:55:51.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:55:51.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:55:51.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:55:51.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:55:51.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:55:51.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2024-03-09T13:55:52.644 INFO:tasks.workunit.client.0.smithi162.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-09T13:55:53.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:55:53.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:55:53.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:55:53.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:55:53.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:55:53.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:55:53.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:55:53.423 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:55:53.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:55:53.424 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:55:53.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:55:53.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:55:53.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:55:53.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:55:53.449 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:53.448+0000 7f710aa0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:53.452 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:53.451+0000 7f710aa0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:53.453 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:53.452+0000 7f710aa0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:55:53.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:55:53.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:55:53.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:55:53.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:55:53.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:55:53.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:53.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:55:53.707 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:55:53.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:53.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:55:53.971 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:53.970+0000 7f710aa0d640 -1 Falling back to public interface 2024-03-09T13:55:53.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:54.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:54.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:54.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:55:54.983 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:55:54.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:54.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:55:55.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:56.252 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:55:56.251+0000 7f710aa0d640 -1 osd.0 148 log_to_monitors true 2024-03-09T13:55:56.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:56.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:56.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:55:56.265 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:55:56.265 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:56.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:55:56.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:55:57.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:55:57.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:55:57.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:55:57.634 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:55:57.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:55:57.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:55:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 152 up_thru 152 down_at 149 last_clean_interval [135,148) [v2:127.0.0.1:6802/3581481988,v1:127.0.0.1:6803/3581481988] [v2:127.0.0.1:6804/3581481988,v1:127.0.0.1:6805/3581481988] exists,up f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:55:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:55:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:55:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:55:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:55:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:55:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:55:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:55:57.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:55:57.959 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:55:57.960 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:55:57.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:55:57.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:55:57.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:55:58.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:55:58.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:55:58.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:55:58.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:55:58.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:55:58.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:55:58.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:55:58.322 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:55:58.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:55:58.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:55:58.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:55:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=652835028994 2024-03-09T13:55:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 652835028994 2024-03-09T13:55:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028994' 2024-03-09T13:55:58.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:55:58.427 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:55:58.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=627065225221 2024-03-09T13:55:58.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 627065225221 2024-03-09T13:55:58.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028994 1-627065225221' 2024-03-09T13:55:58.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:55:58.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-652835028994 2024-03-09T13:55:58.532 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:55:58.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:55:58.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-652835028994 2024-03-09T13:55:58.533 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:55:58.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=652835028994 2024-03-09T13:55:58.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 652835028994' 2024-03-09T13:55:58.535 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 652835028994 2024-03-09T13:55:58.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:55:58.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 652835028994 -lt 652835028994 2024-03-09T13:55:58.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:55:58.842 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-627065225221 2024-03-09T13:55:58.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:55:58.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:55:58.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-627065225221 2024-03-09T13:55:58.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:55:58.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=627065225221 2024-03-09T13:55:58.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 627065225221' 2024-03-09T13:55:58.845 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 627065225221 2024-03-09T13:55:58.845 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:55:59.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 627065225221 -lt 627065225221 2024-03-09T13:55:59.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:55:59.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:55:59.119 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:55:59.459 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:55:59.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:55:59.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:55:59.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:55:59.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:55:59.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:55:59.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:55:59.460 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:55:59.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:55:59.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:55:59.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:55:59.732 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:00.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:56:00.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:56:00.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:56:00.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1118: TEST_corrupt_scrub_replicated: echo -n bad-val 2024-03-09T13:56:00.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1119: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-09T13:56:00.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:56:00.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:56:00.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:56:00.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:56:00.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-09T13:56:00.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:56:00.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:56:00.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:56:00.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:56:00.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:56:00.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:56:00.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:56:00.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:56:00.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:56:00.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:56:00.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:56:00.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-09T13:56:00.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:56:00.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:56:00.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:56:00.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:56:00.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:56:00.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2024-03-09T13:56:01.591 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-09T13:56:01.591 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T13:56:02.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:56:02.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:56:02.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:56:02.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:56:02.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:56:02.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:56:02.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:56:02.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:56:02.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:56:02.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:56:02.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:56:02.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:56:02.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:56:02.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:56:02.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:56:02.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:56:02.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:56:02.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:56:02.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:56:02.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:56:02.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:56:02.115 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:56:02.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:56:02.115 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:56:02.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:56:02.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:56:02.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:56:02.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:56:02.141 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:02.141+0000 7fce5b814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:02.143 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:02.143+0000 7fce5b814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:02.144 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:02.144+0000 7fce5b814640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:02.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:56:02.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:56:02.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:56:02.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:56:02.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:56:02.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:02.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:56:02.397 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:56:02.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:02.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:56:02.663 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:02.663+0000 7fce5b814640 -1 Falling back to public interface 2024-03-09T13:56:02.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:03.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:03.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:03.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:56:03.688 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:56:03.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:03.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:56:03.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:04.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:04.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:04.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:56:04.970 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:56:04.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:04.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:56:04.995 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:04.995+0000 7fce5b814640 -1 osd.1 154 log_to_monitors true 2024-03-09T13:56:05.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:06.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:06.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:06.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:56:06.342 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:56:06.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:06.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:56:06.623 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 158 up_thru 158 down_at 155 last_clean_interval [146,154) [v2:127.0.0.1:6810/3389215104,v1:127.0.0.1:6811/3389215104] [v2:127.0.0.1:6812/3389215104,v1:127.0.0.1:6813/3389215104] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:56:06.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:56:06.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:56:06.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:56:06.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:56:06.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:56:06.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:56:06.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:56:06.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:56:06.624 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:56:06.624 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:56:06.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:56:06.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:56:06.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:56:06.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:56:06.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:56:06.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:56:06.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:56:06.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:56:06.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:56:06.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:56:06.984 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:56:06.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:56:06.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:56:06.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:56:07.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=652835028997 2024-03-09T13:56:07.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 652835028997 2024-03-09T13:56:07.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028997' 2024-03-09T13:56:07.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:56:07.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:56:07.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=678604832770 2024-03-09T13:56:07.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 678604832770 2024-03-09T13:56:07.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-652835028997 1-678604832770' 2024-03-09T13:56:07.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:56:07.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-652835028997 2024-03-09T13:56:07.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:56:07.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:56:07.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-652835028997 2024-03-09T13:56:07.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:56:07.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=652835028997 2024-03-09T13:56:07.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 652835028997' 2024-03-09T13:56:07.198 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 652835028997 2024-03-09T13:56:07.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:56:07.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 652835028996 -lt 652835028997 2024-03-09T13:56:07.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:56:08.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:56:08.474 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:56:08.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 652835028997 -lt 652835028997 2024-03-09T13:56:08.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:56:08.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-678604832770 2024-03-09T13:56:08.757 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:56:08.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:56:08.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-678604832770 2024-03-09T13:56:08.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:56:08.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=678604832770 2024-03-09T13:56:08.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 678604832770' 2024-03-09T13:56:08.760 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 678604832770 2024-03-09T13:56:08.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:56:09.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 678604832770 -lt 678604832770 2024-03-09T13:56:09.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:56:09.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:09.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:09.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:56:09.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:56:09.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:56:09.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:56:09.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:56:09.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:56:09.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:56:09.377 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:56:09.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:56:09.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:56:09.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:09.651 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:09.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:56:09.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:56:09.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:56:09.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:56:09.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ17 2024-03-09T13:56:09.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 17 % 2 2024-03-09T13:56:09.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T13:56:09.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:56:09.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1124: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2024-03-09T13:56:09.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1125: TEST_corrupt_scrub_replicated: echo ROBJ17 2024-03-09T13:56:09.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T13:56:09.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:56:09.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:56:09.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:56:09.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:56:09.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T13:56:09.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:56:09.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:56:09.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:56:09.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:56:09.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:56:09.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:56:09.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:56:09.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:56:09.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:56:09.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:56:10.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:56:10.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T13:56:10.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:56:10.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:56:10.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:56:10.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:56:10.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:56:10.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T13:56:11.478 INFO:tasks.workunit.client.0.smithi162.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-09T13:56:12.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:56:12.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:56:12.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:56:12.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:56:12.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:56:12.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:56:12.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:56:12.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:56:12.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:56:12.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:56:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:56:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:56:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:56:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:56:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:56:12.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:56:12.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:56:12.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:56:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:56:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:56:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:56:12.074 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:56:12.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:56:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:56:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:56:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:56:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:56:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:56:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:56:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:56:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:56:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:56:12.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:56:12.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:56:12.076 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:56:12.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:56:12.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:56:12.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:56:12.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:56:12.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:56:12.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:56:12.102 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:12.102+0000 7faf08c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:12.105 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:12.105+0000 7faf08c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:12.106 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:12.106+0000 7faf08c13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:12.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:56:12.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:56:12.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:56:12.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:56:12.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:56:12.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:12.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:56:12.357 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:56:12.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:12.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:56:12.621 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:12.621+0000 7faf08c13640 -1 Falling back to public interface 2024-03-09T13:56:12.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:13.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:13.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:13.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:56:13.653 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:56:13.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:13.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:56:13.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:14.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:14.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:14.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:56:14.929 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:56:14.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:14.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:56:15.063 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:15.063+0000 7faf08c13640 -1 osd.0 160 log_to_monitors true 2024-03-09T13:56:15.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:16.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:16.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:16.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:56:16.211 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:56:16.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:16.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:56:16.729 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 164 up_thru 164 down_at 161 last_clean_interval [152,160) [v2:127.0.0.1:6802/1449748269,v1:127.0.0.1:6803/1449748269] [v2:127.0.0.1:6804/1449748269,v1:127.0.0.1:6805/1449748269] exists,up f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:56:16.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:56:16.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:56:16.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:56:16.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:56:16.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:56:16.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:56:16.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:56:16.730 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:56:16.730 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:56:16.731 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:56:16.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:56:16.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:56:16.731 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:56:16.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:56:16.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:56:16.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:56:16.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:56:16.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:56:16.819 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:56:17.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:56:17.092 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:56:17.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:56:17.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:56:17.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:56:17.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=704374636546 2024-03-09T13:56:17.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 704374636546 2024-03-09T13:56:17.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636546' 2024-03-09T13:56:17.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:56:17.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:56:17.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=678604832773 2024-03-09T13:56:17.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 678604832773 2024-03-09T13:56:17.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636546 1-678604832773' 2024-03-09T13:56:17.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:56:17.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-704374636546 2024-03-09T13:56:17.301 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:56:17.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:56:17.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-704374636546 2024-03-09T13:56:17.303 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:56:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=704374636546 2024-03-09T13:56:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 704374636546' 2024-03-09T13:56:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 704374636546 2024-03-09T13:56:17.304 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:56:17.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 704374636546 2024-03-09T13:56:17.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:56:18.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:56:18.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:56:18.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 704374636546 -lt 704374636546 2024-03-09T13:56:18.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:56:18.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-678604832773 2024-03-09T13:56:18.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:56:18.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:56:18.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-678604832773 2024-03-09T13:56:18.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:56:18.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=678604832773 2024-03-09T13:56:18.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 678604832773' 2024-03-09T13:56:18.863 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 678604832773 2024-03-09T13:56:18.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:56:19.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 678604832773 -lt 678604832773 2024-03-09T13:56:19.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:56:19.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:19.135 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:19.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:56:19.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:56:19.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:56:19.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:56:19.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:56:19.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:56:19.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:56:19.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:56:19.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:56:19.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:56:19.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:19.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:20.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:56:20.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:56:20.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:56:20.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T13:56:20.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:56:20.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:56:20.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:56:20.086 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:56:20.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T13:56:20.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:56:20.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:56:20.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:56:20.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:56:20.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:56:20.087 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:56:20.087 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:56:20.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:56:20.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:56:20.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:56:20.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:56:20.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T13:56:20.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:56:20.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:56:20.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:56:20.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:56:20.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:56:20.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2024-03-09T13:56:21.327 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-09T13:56:21.327 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T13:56:21.873 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:56:21.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:56:21.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:56:21.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:56:21.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:56:21.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:56:21.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:56:21.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:56:21.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:56:21.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:56:21.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:56:21.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:56:21.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:56:21.876 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:56:21.900 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:21.899+0000 7f02b2d38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:21.902 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:21.902+0000 7f02b2d38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:21.904 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:21.903+0000 7f02b2d38640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:56:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:56:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:56:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:56:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:56:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:22.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:56:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:56:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:22.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:56:22.424 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:22.423+0000 7f02b2d38640 -1 Falling back to public interface 2024-03-09T13:56:22.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:23.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:23.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:23.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:56:23.430 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:56:23.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:23.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:56:23.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:24.503 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:24.501+0000 7f02b2d38640 -1 osd.1 165 log_to_monitors true 2024-03-09T13:56:24.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:24.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:24.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:56:24.712 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:56:24.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:24.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:56:25.015 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 169 up_thru 164 down_at 166 last_clean_interval [158,165) [v2:127.0.0.1:6810/3216009496,v1:127.0.0.1:6811/3216009496] [v2:127.0.0.1:6812/3216009496,v1:127.0.0.1:6813/3216009496] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:56:25.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:56:25.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:56:25.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:56:25.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:56:25.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:56:25.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:56:25.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:56:25.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:56:25.015 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:56:25.015 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:56:25.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:56:25.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:56:25.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:56:25.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:56:25.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:56:25.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:56:25.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:56:25.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:56:25.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:56:25.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:56:25.697 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:56:25.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:56:25.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:56:25.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:56:25.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=704374636549 2024-03-09T13:56:25.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 704374636549 2024-03-09T13:56:25.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636549' 2024-03-09T13:56:25.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:56:25.877 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:56:25.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=725849473026 2024-03-09T13:56:25.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 725849473026 2024-03-09T13:56:25.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-704374636549 1-725849473026' 2024-03-09T13:56:25.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:56:25.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-704374636549 2024-03-09T13:56:25.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:56:25.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:56:25.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-704374636549 2024-03-09T13:56:25.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:56:25.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=704374636549 2024-03-09T13:56:25.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 704374636549' 2024-03-09T13:56:25.984 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 704374636549 2024-03-09T13:56:25.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:56:26.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 704374636547 -lt 704374636549 2024-03-09T13:56:26.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:56:27.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:56:27.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:56:27.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 704374636549 -lt 704374636549 2024-03-09T13:56:27.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:56:27.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-725849473026 2024-03-09T13:56:27.574 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:56:27.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:56:27.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-725849473026 2024-03-09T13:56:27.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:56:27.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=725849473026 2024-03-09T13:56:27.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 725849473026' 2024-03-09T13:56:27.576 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 725849473026 2024-03-09T13:56:27.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:56:27.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 725849473026 -lt 725849473026 2024-03-09T13:56:27.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:56:27.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:27.854 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:28.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:56:28.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:56:28.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:56:28.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:56:28.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:56:28.198 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:56:28.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:56:28.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:56:28.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:56:28.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:56:28.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:28.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:28.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T13:56:28.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T13:56:28.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:56:28.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T13:56:28.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:56:28.815 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:56:28.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:56:28.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:56:28.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:56:28.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:56:28.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:56:28.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:56:28.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:56:28.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:56:29.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:56:29.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:56:29.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:29.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:29.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T13:56:29.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:56:29.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:56:29.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:56:29.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:56:29.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:56:29.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:56:29.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:56:29.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:56:29.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:56:29.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:56:29.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:56:29.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:56:29.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:56:30.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:56:30.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:56:30.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:56:30.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:56:30.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:56:30.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:56:30.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:56:30.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:56:30.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:56:30.337 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:56:30.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:30.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:30.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T13:56:30.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:56:30.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:56:30.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:56:30.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:56:30.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:56:30.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:56:30.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:56:30.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:56:31.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:56:31.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:56:31.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-09T13:56:31.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:56:31.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-09T13:56:31.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:56:31.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:56:31.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:56:31.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:56:31.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:56:31.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:56:31.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:56:31.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:56:31.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:56:31.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:56:31.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:31.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:32.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:56:32.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:56:32.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:56:32.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:56:32.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ18 2024-03-09T13:56:32.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 18 % 2 2024-03-09T13:56:32.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=0 2024-03-09T13:56:32.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:56:32.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1132: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2024-03-09T13:56:32.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo ROBJ18 2024-03-09T13:56:32.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T13:56:32.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:56:32.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:56:32.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:56:32.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:56:32.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T13:56:32.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:56:32.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:56:32.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:56:32.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:56:32.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:56:32.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:56:32.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:56:32.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:56:32.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:56:32.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:56:32.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:56:32.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T13:56:32.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:56:32.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:56:32.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:56:32.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:56:32.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:56:32.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T13:56:33.389 INFO:tasks.workunit.client.0.smithi162.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-09T13:56:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:56:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:56:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:56:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:56:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:56:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:56:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:56:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:56:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:56:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:56:33.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:56:33.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:56:33.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:56:33.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:56:33.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:56:33.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:56:33.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:56:33.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:56:33.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:56:33.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:56:33.985 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:56:33.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:56:33.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:56:33.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:56:33.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:56:33.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:56:33.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:56:34.015 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:34.014+0000 7f9d4fc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:34.017 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:34.016+0000 7f9d4fc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:34.018 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:34.017+0000 7f9d4fc0d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:34.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:56:34.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:56:34.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:56:34.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:56:34.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:56:34.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:34.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:56:34.269 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:56:34.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:34.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:56:34.534 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:34.533+0000 7f9d4fc0d640 -1 Falling back to public interface 2024-03-09T13:56:34.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:35.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:35.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:35.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:56:35.543 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:56:35.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:35.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:56:35.817 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:36.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:36.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:36.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:56:36.819 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:56:36.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:36.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:56:36.980 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:36.979+0000 7f9d4fc0d640 -1 osd.0 171 log_to_monitors true 2024-03-09T13:56:37.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:38.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:38.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:38.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:56:38.104 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:56:38.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:38.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:56:38.389 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 175 up_thru 175 down_at 172 last_clean_interval [164,171) [v2:127.0.0.1:6802/914122002,v1:127.0.0.1:6803/914122002] [v2:127.0.0.1:6804/914122002,v1:127.0.0.1:6805/914122002] exists,up f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:56:38.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:56:38.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:56:38.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:56:38.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:56:38.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:56:38.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:56:38.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:56:38.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:56:38.390 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:56:38.390 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:56:38.390 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:56:38.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:56:38.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:56:38.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:56:38.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:56:38.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:56:38.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:56:38.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:56:38.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:56:38.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:56:38.776 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:56:38.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:56:38.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:56:38.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:56:38.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=751619276802 2024-03-09T13:56:38.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 751619276802 2024-03-09T13:56:38.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276802' 2024-03-09T13:56:38.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:56:38.881 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:56:38.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=725849473029 2024-03-09T13:56:38.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 725849473029 2024-03-09T13:56:38.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276802 1-725849473029' 2024-03-09T13:56:38.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:56:38.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-751619276802 2024-03-09T13:56:38.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:56:38.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:56:38.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-751619276802 2024-03-09T13:56:38.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:56:38.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=751619276802 2024-03-09T13:56:38.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 751619276802' 2024-03-09T13:56:38.988 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 751619276802 2024-03-09T13:56:38.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:56:39.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 751619276802 2024-03-09T13:56:39.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:56:40.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:56:40.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:56:40.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 751619276802 2024-03-09T13:56:40.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:56:41.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:56:41.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:56:41.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276802 -lt 751619276802 2024-03-09T13:56:41.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:56:41.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-725849473029 2024-03-09T13:56:41.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:56:41.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:56:41.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-725849473029 2024-03-09T13:56:41.826 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:56:41.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=725849473029 2024-03-09T13:56:41.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 725849473029' 2024-03-09T13:56:41.827 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 725849473029 2024-03-09T13:56:41.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:56:42.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 725849473030 -lt 725849473029 2024-03-09T13:56:42.102 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:56:42.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:42.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:42.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:56:42.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:56:42.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:56:42.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:56:42.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:56:42.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:56:42.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:56:42.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:56:42.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:56:42.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:56:42.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:42.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:43.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:56:43.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:56:43.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:56:43.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T13:56:43.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:56:43.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:56:43.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:56:43.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:56:43.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T13:56:43.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:56:43.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:56:43.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:56:43.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:56:43.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:56:43.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:56:43.059 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:56:43.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:56:43.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:56:43.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:56:43.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:56:43.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T13:56:43.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:56:43.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:56:43.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:56:43.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:56:43.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:56:43.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2024-03-09T13:56:44.232 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-09T13:56:44.232 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T13:56:44.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:56:44.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:56:44.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:56:44.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:56:44.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:56:44.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:56:44.780 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:56:44.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:56:44.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:56:44.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:56:44.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:56:44.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:56:44.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:56:44.805 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:44.804+0000 7fd74f80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:44.808 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:44.807+0000 7fd74f80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:44.809 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:44.808+0000 7fd74f80c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:45.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:56:45.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:56:45.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:56:45.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:56:45.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:56:45.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:45.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:56:45.061 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:56:45.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:45.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:56:45.324 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:45.324+0000 7fd74f80c640 -1 Falling back to public interface 2024-03-09T13:56:45.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:46.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:46.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:46.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:56:46.332 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:56:46.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:46.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:56:46.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:47.414 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:47.414+0000 7fd74f80c640 -1 osd.1 177 log_to_monitors true 2024-03-09T13:56:47.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:47.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:47.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:56:47.615 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:56:47.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:47.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:56:48.016 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 181 up_thru 176 down_at 178 last_clean_interval [169,177) [v2:127.0.0.1:6810/3532292132,v1:127.0.0.1:6811/3532292132] [v2:127.0.0.1:6812/3532292132,v1:127.0.0.1:6813/3532292132] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:56:48.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:56:48.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:56:48.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:56:48.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:56:48.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:56:48.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:56:48.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:56:48.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:56:48.017 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:56:48.017 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:56:48.017 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:56:48.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:56:48.018 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:56:48.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:56:48.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:56:48.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:56:48.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:56:48.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:56:48.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:56:48.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:56:48.443 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:56:48.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:56:48.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:56:48.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:56:48.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=751619276805 2024-03-09T13:56:48.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 751619276805 2024-03-09T13:56:48.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276805' 2024-03-09T13:56:48.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:56:48.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:56:48.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=777389080578 2024-03-09T13:56:48.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 777389080578 2024-03-09T13:56:48.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276805 1-777389080578' 2024-03-09T13:56:48.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:56:48.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-751619276805 2024-03-09T13:56:48.837 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:56:48.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:56:48.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-751619276805 2024-03-09T13:56:48.838 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:56:48.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=751619276805 2024-03-09T13:56:48.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 751619276805' 2024-03-09T13:56:48.840 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 751619276805 2024-03-09T13:56:48.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:56:49.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276804 -lt 751619276805 2024-03-09T13:56:49.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:56:50.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:56:50.137 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:56:50.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276804 -lt 751619276805 2024-03-09T13:56:50.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:56:51.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:56:51.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:56:51.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276805 -lt 751619276805 2024-03-09T13:56:51.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:56:51.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-777389080578 2024-03-09T13:56:51.703 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:56:51.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:56:51.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-777389080578 2024-03-09T13:56:51.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:56:51.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=777389080578 2024-03-09T13:56:51.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 777389080578' 2024-03-09T13:56:51.706 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 777389080578 2024-03-09T13:56:51.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:56:51.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 777389080578 -lt 777389080578 2024-03-09T13:56:51.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:56:51.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:51.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:52.323 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:56:52.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:56:52.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:56:52.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:56:52.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:56:52.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:56:52.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:56:52.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:56:52.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:56:52.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:56:52.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:52.595 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:52.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:56:52.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:56:52.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:56:52.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1137: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-09T13:56:52.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:56:52.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:56:52.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:56:52.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:56:52.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-09T13:56:52.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:56:52.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:56:52.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:56:52.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:56:52.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:56:52.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:56:52.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:56:52.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:56:52.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:56:52.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:56:53.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:56:53.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2024-03-09T13:56:53.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:56:53.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:56:53.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:56:53.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:56:53.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:56:53.261 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2024-03-09T13:56:54.089 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-09T13:56:54.089 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T13:56:54.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:56:54.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:56:54.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:56:54.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:56:54.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:56:54.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:56:54.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:56:54.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:56:54.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:56:54.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:56:54.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:56:54.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:56:54.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:56:54.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:56:54.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:56:54.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:56:54.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:56:54.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:56:54.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:56:54.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:56:54.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:56:54.609 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:56:54.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:56:54.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:56:54.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:56:54.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:56:54.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:56:54.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:56:54.636 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:54.636+0000 7f66fca68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:54.646 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:54.646+0000 7f66fca68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:54.648 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:54.648+0000 7f66fca68640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:56:54.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:56:54.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:56:54.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:56:54.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:56:54.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:56:54.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:54.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:56:54.893 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:56:54.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:54.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:56:55.164 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:55.164+0000 7f66fca68640 -1 Falling back to public interface 2024-03-09T13:56:55.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:56.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:56.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:56.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:56:56.166 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:56:56.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:56.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:56:56.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:56:57.265 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:56:57.265+0000 7f66fca68640 -1 osd.1 183 log_to_monitors true 2024-03-09T13:56:57.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:56:57.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:56:57.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:56:57.452 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:56:57.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:56:57.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:56:57.738 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 187 up_thru 181 down_at 184 last_clean_interval [181,183) [v2:127.0.0.1:6810/3175660672,v1:127.0.0.1:6811/3175660672] [v2:127.0.0.1:6812/3175660672,v1:127.0.0.1:6813/3175660672] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:56:57.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:56:57.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:56:57.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:56:57.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:56:57.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:56:57.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:56:57.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:56:57.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:56:57.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:56:57.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:56:57.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:56:57.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:56:57.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:56:57.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:56:57.827 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:56:57.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:56:57.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:56:57.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:56:57.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:56:58.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:56:58.104 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:56:58.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:56:58.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:56:58.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:56:58.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=751619276808 2024-03-09T13:56:58.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 751619276808 2024-03-09T13:56:58.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276808' 2024-03-09T13:56:58.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:56:58.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:56:58.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=803158884354 2024-03-09T13:56:58.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 803158884354 2024-03-09T13:56:58.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-751619276808 1-803158884354' 2024-03-09T13:56:58.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:56:58.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-751619276808 2024-03-09T13:56:58.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:56:58.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:56:58.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-751619276808 2024-03-09T13:56:58.476 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:56:58.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=751619276808 2024-03-09T13:56:58.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 751619276808' 2024-03-09T13:56:58.478 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 751619276808 2024-03-09T13:56:58.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:56:58.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 751619276808 -lt 751619276808 2024-03-09T13:56:58.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:56:58.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-803158884354 2024-03-09T13:56:58.799 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:56:58.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:56:58.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-803158884354 2024-03-09T13:56:58.800 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:56:58.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=803158884354 2024-03-09T13:56:58.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 803158884354' 2024-03-09T13:56:58.802 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 803158884354 2024-03-09T13:56:58.802 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:56:59.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 803158884354 -lt 803158884354 2024-03-09T13:56:59.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:56:59.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:59.077 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:56:59.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:56:59.416 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:56:59.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:56:59.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:56:59.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:56:59.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:56:59.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:56:59.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:56:59.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:56:59.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:56:59.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:56:59.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:00.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T13:57:00.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T13:57:00.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:57:00.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T13:57:00.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:57:00.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:57:00.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:57:00.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:57:00.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:57:00.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:57:00.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:57:00.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:57:00.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:57:00.126 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:57:00.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:57:00.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:57:00.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:00.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:00.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T13:57:00.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:57:00.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:57:00.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:57:00.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:57:00.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:57:00.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:57:00.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:57:00.766 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:57:01.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:57:01.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:57:01.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:57:01.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:57:01.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:57:01.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:57:01.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:57:01.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:57:01.309 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:57:01.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:57:01.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:57:01.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:57:01.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:57:01.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:57:01.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:57:01.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:01.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:01.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T13:57:01.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:57:01.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:57:01.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:57:01.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:57:01.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:57:01.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:57:01.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:57:01.921 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:57:02.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:57:02.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:57:02.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-09T13:57:02.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:57:02.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-09T13:57:02.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:57:02.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:57:02.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:57:02.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:57:02.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:57:02.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:57:02.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:57:02.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:57:02.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:57:02.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:57:02.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:02.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:03.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:57:03.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:57:03.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:57:03.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2024-03-09T13:57:03.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: TEST_corrupt_scrub_replicated: objname=ROBJ19 2024-03-09T13:57:03.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: expr 19 % 2 2024-03-09T13:57:03.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1033: TEST_corrupt_scrub_replicated: local osd=1 2024-03-09T13:57:03.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1035: TEST_corrupt_scrub_replicated: case $i in 2024-03-09T13:57:03.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2024-03-09T13:57:03.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-09T13:57:03.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-09T13:57:03.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-09T13:57:03.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:57:03.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: local pg=3.0 2024-03-09T13:57:03.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2024-03-09T13:57:03.662 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0: {} 2024-03-09T13:57:03.663 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-03-09T13:57:03.679 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1: {} 2024-03-09T13:57:03.680 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-03-09T13:57:03.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1150: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-03-09T13:57:03.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T13:57:03.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T13:57:03.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-09T13:57:03.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T13:57:03.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T13:57:03.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T13:57:03.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-03-09T13:57:03.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T13:57:03.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T13:57:03.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T13:57:03.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-09T13:57:03.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T13:57:03.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-03-09T13:57:03.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T13:57:03.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-03-09T13:57:03.691 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-03-09T13:57:03.691 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T13:57:03.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T13:57:03.981 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T13:57:03.981 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T13:57:03.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T13:57:03.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T13:57:03.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-09T13:57:03.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T13:57:03.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T13:57:03.982 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-09T13:57:03.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T13:57:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-09T13:57:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T13:57:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-09T13:57:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T13:57:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T13:57:03.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T13:57:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-09T13:57:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T13:57:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T13:57:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:03.984 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.1.asok 2024-03-09T13:57:03.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70042/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-09T13:57:04.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T13:57:04.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T13:57:04.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T13:57:04.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-09T13:57:04.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T13:57:04.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T13:57:04.073 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:04.073 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:04.073 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:04.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.1.asok 2024-03-09T13:57:04.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T13:57:04.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70042/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-03-09T13:57:04.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1151: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-03-09T13:57:04.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T13:57:04.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T13:57:04.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-09T13:57:04.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T13:57:04.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T13:57:04.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T13:57:04.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-03-09T13:57:04.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T13:57:04.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T13:57:04.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T13:57:04.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-09T13:57:04.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T13:57:04.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-03-09T13:57:04.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T13:57:04.163 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-03-09T13:57:04.163 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-03-09T13:57:04.163 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T13:57:04.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T13:57:04.438 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T13:57:04.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T13:57:04.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T13:57:04.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T13:57:04.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-09T13:57:04.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T13:57:04.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T13:57:04.440 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-09T13:57:04.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T13:57:04.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-09T13:57:04.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T13:57:04.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-09T13:57:04.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T13:57:04.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T13:57:04.441 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T13:57:04.441 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-09T13:57:04.442 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T13:57:04.442 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T13:57:04.442 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:04.442 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:04.442 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:04.442 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.0.asok 2024-03-09T13:57:04.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70042/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-09T13:57:04.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T13:57:04.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T13:57:04.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T13:57:04.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-09T13:57:04.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T13:57:04.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T13:57:04.528 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:04.528 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:04.528 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:04.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.0.asok 2024-03-09T13:57:04.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T13:57:04.528 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70042/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-03-09T13:57:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-03-09T13:57:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T13:57:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T13:57:04.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-09T13:57:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T13:57:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T13:57:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T13:57:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-09T13:57:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T13:57:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T13:57:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T13:57:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-09T13:57:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T13:57:04.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-09T13:57:04.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T13:57:04.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-09T13:57:04.618 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-09T13:57:04.619 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T13:57:04.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T13:57:04.903 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T13:57:04.903 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T13:57:04.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T13:57:04.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T13:57:04.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-09T13:57:04.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T13:57:04.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T13:57:04.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-09T13:57:04.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T13:57:04.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-09T13:57:04.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T13:57:04.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-09T13:57:04.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T13:57:04.904 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T13:57:04.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T13:57:04.905 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-09T13:57:04.905 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T13:57:04.905 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T13:57:04.905 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:04.905 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:04.905 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:04.905 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.0.asok 2024-03-09T13:57:04.905 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70042/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-09T13:57:04.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T13:57:04.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T13:57:04.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T13:57:04.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-09T13:57:04.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T13:57:04.991 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T13:57:04.991 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:04.992 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:04.992 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:04.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.0.asok 2024-03-09T13:57:04.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T13:57:04.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70042/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-03-09T13:57:05.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-03-09T13:57:05.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T13:57:05.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T13:57:05.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-09T13:57:05.080 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T13:57:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T13:57:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T13:57:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-09T13:57:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T13:57:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T13:57:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T13:57:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-09T13:57:05.081 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T13:57:05.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-09T13:57:05.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T13:57:05.081 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-09T13:57:05.082 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-09T13:57:05.082 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T13:57:05.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T13:57:05.358 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T13:57:05.358 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T13:57:05.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T13:57:05.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T13:57:05.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-09T13:57:05.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T13:57:05.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T13:57:05.359 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-09T13:57:05.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.1.asok 2024-03-09T13:57:05.361 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70042/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-09T13:57:05.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T13:57:05.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T13:57:05.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T13:57:05.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-09T13:57:05.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T13:57:05.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T13:57:05.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:05.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:05.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:05.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.1.asok 2024-03-09T13:57:05.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T13:57:05.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70042/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-03-09T13:57:05.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2024-03-09T13:57:05.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-09T13:57:05.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-09T13:57:05.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:57:05.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:57:05.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:57:05.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:57:05.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-09T13:53:15.768738+0000 2024-03-09T13:57:05.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-09T13:57:06.065 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 3.0 on osd.1 to scrub 2024-03-09T13:57:06.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-09T13:53:15.768738+0000 2024-03-09T13:57:06.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T13:57:06.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:53:15.768738+0000 2024-03-09T13:57:06.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:57:06.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:57:06.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:57:06.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:57:06.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:57:06.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:57:06.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:57:06.078 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:57:06.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:53:15.768738+0000 '>' 2024-03-09T13:53:15.768738+0000 2024-03-09T13:57:06.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:57:07.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:57:07.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:57:07.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:57:07.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:57:07.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:57:07.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:57:07.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:57:07.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:53:15.768738+0000 '>' 2024-03-09T13:53:15.768738+0000 2024-03-09T13:57:07.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:57:08.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:57:08.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:57:08.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:57:08.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:57:08.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:57:08.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:57:08.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:57:09.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:57:06.313143+0000 '>' 2024-03-09T13:53:15.768738+0000 2024-03-09T13:57:09.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:57:09.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1157: TEST_corrupt_scrub_replicated: ERRORS=0 2024-03-09T13:57:09.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1158: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2024-03-09T13:57:09.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1159: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-03-09T13:57:09.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1160: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2024-03-09T13:57:09.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1161: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-03-09T13:57:09.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1162: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-03-09T13:57:09.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1163: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1164: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1169: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1170: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1171: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1173: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1174: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1175: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-03-09T13:57:09.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-09T13:57:09.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2024-03-09T13:57:09.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1179: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2024-03-09T13:57:09.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1180: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2024-03-09T13:57:09.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.874 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:09.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:09.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:10.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:10.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:10.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:10.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:10.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1182: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:57:10.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2024-03-09T13:57:10.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-03-09T13:57:10.061 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-03-09T13:57:10.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: test 1 = 1 2024-03-09T13:57:10.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T13:57:10.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1195: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-03-09T13:57:10.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1197: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-03-09T13:57:10.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-03-09T13:57:10.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1199: TEST_corrupt_scrub_replicated: epoch=187 2024-03-09T13:57:10.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-09T13:57:10.096 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T13:57:10.096 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T13:57:10.096 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T13:57:10.096 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T13:57:10.096 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T13:57:10.096 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T13:57:10.096 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T13:57:10.096 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T13:57:10.097 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T13:57:10.097 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T13:57:10.097 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T13:57:10.097 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T13:57:10.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-09T13:57:10.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1201: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T13:57:10.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-09T13:57:10.120 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T13:57:10.120 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T13:57:10.120 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T13:57:10.120 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T13:57:10.120 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T13:57:10.120 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T13:57:10.120 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T13:57:10.120 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T13:57:10.120 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T13:57:10.120 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T13:57:10.120 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T13:57:10.120 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T13:57:10.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-09T13:57:10.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2082: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T13:57:10.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2083: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T13:57:10.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T13:57:10.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2084: TEST_corrupt_scrub_replicated: test no = yes 2024-03-09T13:57:10.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2089: TEST_corrupt_scrub_replicated: test '' = yes 2024-03-09T13:57:10.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2094: TEST_corrupt_scrub_replicated: objname=ROBJ9 2024-03-09T13:57:10.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2096: TEST_corrupt_scrub_replicated: echo -n ZZZ 2024-03-09T13:57:10.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2097: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2024-03-09T13:57:10.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T13:57:10.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:57:10.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:57:10.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:57:10.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:57:10.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T13:57:10.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:57:10.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:57:10.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:57:10.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:57:10.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:57:10.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:57:10.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:57:10.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:57:10.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:57:10.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:57:10.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:57:10.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T13:57:10.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:57:10.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:57:10.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:57:10.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:57:10.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:57:10.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2024-03-09T13:57:11.423 INFO:tasks.workunit.client.0.smithi162.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-09T13:57:11.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:57:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:57:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:57:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:57:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:57:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:57:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:57:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:57:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:57:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:57:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:57:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:57:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:57:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:57:11.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:57:11.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:57:11.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:57:11.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:57:11.941 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:57:11.941 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:11.941 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:11.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:57:11.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:57:11.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:57:11.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:57:11.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:57:11.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:57:11.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:57:11.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:57:11.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:57:11.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:57:11.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:57:11.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:57:11.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:57:11.943 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:57:11.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:57:11.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:57:11.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:57:11.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:57:11.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:57:11.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:57:11.971 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:11.970+0000 7fe608e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:57:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:11.973+0000 7fe608e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:57:11.975 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:11.974+0000 7fe608e25640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:57:12.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:57:12.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:57:12.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:57:12.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:57:12.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:57:12.226 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:12.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:57:12.227 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:57:12.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:12.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:57:12.489 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:12.488+0000 7fe608e25640 -1 Falling back to public interface 2024-03-09T13:57:12.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:57:13.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:57:13.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:13.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:57:13.502 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:57:13.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:13.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:57:13.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:57:14.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:57:14.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:14.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:57:14.781 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:57:14.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:14.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:57:15.001 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:15.001+0000 7fe608e25640 -1 osd.0 189 log_to_monitors true 2024-03-09T13:57:15.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:57:16.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:57:16.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:16.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:57:16.113 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:57:16.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:16.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:57:16.402 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 193 up_thru 193 down_at 190 last_clean_interval [175,189) [v2:127.0.0.1:6802/59878806,v1:127.0.0.1:6803/59878806] [v2:127.0.0.1:6804/59878806,v1:127.0.0.1:6805/59878806] exists,up f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:57:16.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:57:16.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:57:16.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:57:16.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:57:16.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:57:16.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:57:16.403 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:57:16.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:57:16.403 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:57:16.403 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:57:16.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:57:16.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:57:16.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:57:16.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:57:16.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:57:16.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:57:16.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:57:16.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:57:16.491 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:57:16.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:57:16.764 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:57:16.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:57:16.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:57:16.764 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:57:16.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=828928688130 2024-03-09T13:57:16.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 828928688130 2024-03-09T13:57:16.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-828928688130' 2024-03-09T13:57:16.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:57:16.888 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:57:16.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=803158884358 2024-03-09T13:57:16.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 803158884358 2024-03-09T13:57:16.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-828928688130 1-803158884358' 2024-03-09T13:57:16.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:57:16.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-828928688130 2024-03-09T13:57:16.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:57:16.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:57:16.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-828928688130 2024-03-09T13:57:16.993 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:57:16.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=828928688130 2024-03-09T13:57:16.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 828928688130' 2024-03-09T13:57:16.995 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 828928688130 2024-03-09T13:57:16.995 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:57:17.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 828928688130 2024-03-09T13:57:17.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:57:18.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:57:18.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:57:18.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 828928688130 2024-03-09T13:57:18.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:57:19.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:57:19.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:57:19.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 828928688130 -lt 828928688130 2024-03-09T13:57:19.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:57:19.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-803158884358 2024-03-09T13:57:19.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:57:19.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:57:19.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-803158884358 2024-03-09T13:57:19.832 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:57:19.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=803158884358 2024-03-09T13:57:19.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 803158884358' 2024-03-09T13:57:19.833 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 803158884358 2024-03-09T13:57:19.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:57:20.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 803158884359 -lt 803158884358 2024-03-09T13:57:20.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:57:20.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:20.105 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:20.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:57:20.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:57:20.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:57:20.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:57:20.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:57:20.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:57:20.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:57:20.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:57:20.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:57:20.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:57:20.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:20.723 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:21.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:57:21.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:57:21.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:57:21.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2100: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2024-03-09T13:57:21.064 INFO:tasks.workunit.client.0.smithi162.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2024-03-09T13:57:21.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2102: TEST_corrupt_scrub_replicated: objname=ROBJ10 2024-03-09T13:57:21.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2103: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-09T13:57:21.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:57:21.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:57:21.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:57:21.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:57:21.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-09T13:57:21.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:57:21.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:57:21.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:57:21.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:57:21.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:57:21.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:57:21.065 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:57:21.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:57:21.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:57:21.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:57:21.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:57:21.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2024-03-09T13:57:21.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:57:21.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:57:21.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:57:21.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:57:21.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:57:21.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2024-03-09T13:57:22.193 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-09T13:57:22.193 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T13:57:22.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:57:22.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:57:22.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:57:22.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:57:22.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:57:22.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:57:22.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:57:22.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:57:22.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:57:22.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:57:22.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:57:22.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:57:22.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:57:22.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:57:22.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:57:22.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:57:22.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:57:22.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:57:22.715 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:57:22.715 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:22.715 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:22.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:57:22.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:57:22.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:57:22.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:57:22.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:57:22.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:57:22.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:57:22.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:57:22.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:57:22.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:57:22.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:57:22.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:57:22.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:57:22.716 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:57:22.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:57:22.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:57:22.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:57:22.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:57:22.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:57:22.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:57:22.742 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:22.741+0000 7fd5b1cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:57:22.763 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:22.762+0000 7fd5b1cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:57:22.764 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:22.763+0000 7fd5b1cb5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:57:22.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:57:22.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:57:22.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:57:22.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:57:22.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:57:22.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:22.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:57:22.996 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:57:22.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:22.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:57:23.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:57:23.286 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:23.286+0000 7fd5b1cb5640 -1 Falling back to public interface 2024-03-09T13:57:24.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:57:24.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:24.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:57:24.269 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:57:24.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:24.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:57:24.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:57:25.360 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:25.359+0000 7fd5b1cb5640 -1 osd.1 195 log_to_monitors true 2024-03-09T13:57:25.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:57:25.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:25.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:57:25.548 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:57:25.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:25.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:57:26.030 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 199 up_thru 194 down_at 196 last_clean_interval [187,195) [v2:127.0.0.1:6810/538281302,v1:127.0.0.1:6811/538281302] [v2:127.0.0.1:6812/538281302,v1:127.0.0.1:6813/538281302] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:57:26.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:57:26.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:57:26.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:57:26.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:57:26.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:57:26.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:57:26.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:57:26.030 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:57:26.031 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:57:26.031 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:57:26.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:57:26.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:57:26.031 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:57:26.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:57:26.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:57:26.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:57:26.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:57:26.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:57:26.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:57:26.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:57:26.551 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:57:26.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:57:26.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:57:26.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:57:26.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=828928688133 2024-03-09T13:57:26.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 828928688133 2024-03-09T13:57:26.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-828928688133' 2024-03-09T13:57:26.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:57:26.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:57:26.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=854698491906 2024-03-09T13:57:26.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 854698491906 2024-03-09T13:57:26.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-828928688133 1-854698491906' 2024-03-09T13:57:26.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:57:26.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-828928688133 2024-03-09T13:57:26.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:57:26.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:57:26.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-828928688133 2024-03-09T13:57:26.806 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:57:26.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=828928688133 2024-03-09T13:57:26.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 828928688133' 2024-03-09T13:57:26.807 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 828928688133 2024-03-09T13:57:26.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:57:27.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 828928688132 -lt 828928688133 2024-03-09T13:57:27.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:57:28.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:57:28.085 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:57:28.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 828928688132 -lt 828928688133 2024-03-09T13:57:28.369 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:57:29.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T13:57:29.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:57:29.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 828928688133 -lt 828928688133 2024-03-09T13:57:29.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:57:29.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-854698491906 2024-03-09T13:57:29.653 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:57:29.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:57:29.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-854698491906 2024-03-09T13:57:29.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:57:29.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=854698491906 2024-03-09T13:57:29.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 854698491906' 2024-03-09T13:57:29.656 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 854698491906 2024-03-09T13:57:29.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:57:29.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 854698491906 -lt 854698491906 2024-03-09T13:57:29.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:57:29.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:29.932 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:30.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:57:30.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:57:30.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:57:30.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:57:30.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:57:30.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:57:30.276 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:57:30.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:57:30.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:57:30.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:57:30.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:30.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:30.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:57:30.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:57:30.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:57:30.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2104: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2024-03-09T13:57:31.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2105: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T13:57:31.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:57:31.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:57:31.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:57:31.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:57:31.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T13:57:31.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:57:31.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:57:31.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:57:31.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:57:31.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:57:31.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:57:31.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:57:31.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:57:31.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:57:31.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:57:31.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:57:31.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T13:57:31.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:57:31.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:57:31.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:57:31.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:57:31.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:57:31.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T13:57:32.444 INFO:tasks.workunit.client.0.smithi162.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2024-03-09T13:57:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:57:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:57:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:57:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:57:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:57:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:57:32.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:32.963 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:32.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:57:32.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:57:32.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:57:32.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:57:32.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:57:32.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:57:32.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:57:32.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:57:32.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:57:32.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:57:32.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:57:32.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:57:32.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:57:32.965 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:57:32.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:57:32.965 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:57:32.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:57:32.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:57:32.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:57:32.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:57:32.992 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:32.991+0000 7f9393011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:57:32.994 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:32.993+0000 7f9393011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:57:32.995 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:32.995+0000 7f9393011640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:57:33.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:57:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:57:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:57:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:57:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:57:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:57:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:57:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:33.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:57:33.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:57:33.521 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:33.521+0000 7f9393011640 -1 Falling back to public interface 2024-03-09T13:57:34.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:57:34.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:34.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:57:34.520 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:57:34.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:34.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:57:34.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:57:35.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:57:35.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:35.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:57:35.807 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:57:35.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:35.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:57:35.900 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:35.900+0000 7f9393011640 -1 osd.0 201 log_to_monitors true 2024-03-09T13:57:36.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:57:37.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:57:37.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:37.114 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:57:37.114 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:57:37.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:37.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:57:37.399 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 205 up_thru 205 down_at 202 last_clean_interval [193,201) [v2:127.0.0.1:6802/2179298490,v1:127.0.0.1:6803/2179298490] [v2:127.0.0.1:6804/2179298490,v1:127.0.0.1:6805/2179298490] exists,up f88b04f5-8017-47a7-91b8-f6d2e01cd1ea 2024-03-09T13:57:37.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:57:37.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:57:37.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:57:37.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:57:37.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:57:37.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:57:37.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:57:37.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:57:37.400 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:57:37.400 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:57:37.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:57:37.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:57:37.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:57:37.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:57:37.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:57:37.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:57:37.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:57:37.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:57:37.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:57:37.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:57:37.787 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:57:37.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:57:37.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:57:37.787 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:57:37.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=880468295682 2024-03-09T13:57:37.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 880468295682 2024-03-09T13:57:37.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295682' 2024-03-09T13:57:37.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:57:37.891 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:57:37.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=854698491909 2024-03-09T13:57:37.993 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 854698491909 2024-03-09T13:57:37.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295682 1-854698491909' 2024-03-09T13:57:37.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:57:37.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-880468295682 2024-03-09T13:57:37.994 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:57:37.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:57:37.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-880468295682 2024-03-09T13:57:37.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:57:37.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=880468295682 2024-03-09T13:57:37.997 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 880468295682' 2024-03-09T13:57:37.997 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 880468295682 2024-03-09T13:57:37.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:57:38.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 880468295682 2024-03-09T13:57:38.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:57:39.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:57:39.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:57:39.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295682 -lt 880468295682 2024-03-09T13:57:39.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:57:39.554 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-854698491909 2024-03-09T13:57:39.555 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:57:39.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:57:39.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-854698491909 2024-03-09T13:57:39.556 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:57:39.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=854698491909 2024-03-09T13:57:39.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 854698491909' 2024-03-09T13:57:39.558 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 854698491909 2024-03-09T13:57:39.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:57:39.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 854698491909 -lt 854698491909 2024-03-09T13:57:39.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:57:39.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:39.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:40.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:57:40.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:57:40.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:57:40.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:57:40.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:57:40.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:57:40.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:57:40.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:57:40.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:57:40.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:57:40.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:40.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:40.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:57:40.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:57:40.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:57:40.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T13:57:40.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:57:40.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:57:40.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:57:40.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:57:40.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T13:57:40.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:57:40.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:57:40.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:57:40.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:57:40.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:57:40.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:57:40.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:57:40.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:57:40.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:57:40.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:57:41.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:57:41.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T13:57:41.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:57:41.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:57:41.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:57:41.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:57:41.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:57:41.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2024-03-09T13:57:42.009 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2024-03-09T13:57:42.010 INFO:tasks.workunit.client.0.smithi162.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2024-03-09T13:57:42.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:57:42.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:57:42.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:57:42.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:57:42.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:57:42.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:57:42.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:57:42.536 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:57:42.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:57:42.537 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:57:42.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:57:42.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:57:42.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:57:42.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:57:42.564 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:42.564+0000 7fa44d811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:57:42.566 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:42.566+0000 7fa44d811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:57:42.567 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:42.568+0000 7fa44d811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:57:42.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:57:42.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:57:42.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:57:42.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:57:42.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:57:42.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:42.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:57:42.818 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:57:42.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:42.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:57:43.086 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:43.086+0000 7fa44d811640 -1 Falling back to public interface 2024-03-09T13:57:43.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:57:44.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:57:44.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:44.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:57:44.123 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:57:44.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:44.123 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:57:44.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:57:45.167 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:57:45.167+0000 7fa44d811640 -1 osd.1 207 log_to_monitors true 2024-03-09T13:57:45.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:57:45.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:57:45.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:57:45.408 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:57:45.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:57:45.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:57:45.838 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 211 up_thru 206 down_at 208 last_clean_interval [199,207) [v2:127.0.0.1:6810/1893768448,v1:127.0.0.1:6811/1893768448] [v2:127.0.0.1:6812/1893768448,v1:127.0.0.1:6813/1893768448] exists,up 5c20d4d1-92ce-4b8d-8dab-b9b376ea020c 2024-03-09T13:57:45.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:57:45.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:57:45.838 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:57:45.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:57:45.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:57:45.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:57:45.839 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:57:45.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:57:45.839 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:57:45.839 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:57:45.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:57:45.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:57:45.839 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:57:45.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:57:45.928 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:57:45.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:57:45.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:57:45.929 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:57:45.929 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:57:46.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:57:46.204 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:57:46.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:57:46.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:57:46.205 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:57:46.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=880468295685 2024-03-09T13:57:46.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 880468295685 2024-03-09T13:57:46.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295685' 2024-03-09T13:57:46.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:57:46.347 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:57:46.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=906238099458 2024-03-09T13:57:46.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 906238099458 2024-03-09T13:57:46.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295685 1-906238099458' 2024-03-09T13:57:46.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:57:46.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-880468295685 2024-03-09T13:57:46.519 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:57:46.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:57:46.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-880468295685 2024-03-09T13:57:46.520 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:57:46.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=880468295685 2024-03-09T13:57:46.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 880468295685' 2024-03-09T13:57:46.521 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 880468295685 2024-03-09T13:57:46.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:57:46.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295685 -lt 880468295685 2024-03-09T13:57:46.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:57:46.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-906238099458 2024-03-09T13:57:46.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:57:46.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:57:46.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-906238099458 2024-03-09T13:57:46.797 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:57:46.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=906238099458 2024-03-09T13:57:46.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 906238099458' 2024-03-09T13:57:46.798 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 906238099458 2024-03-09T13:57:46.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:57:47.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 906238099458 -lt 906238099458 2024-03-09T13:57:47.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:57:47.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:47.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:47.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:57:47.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:57:47.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:57:47.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:57:47.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:57:47.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:57:47.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:57:47.413 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:57:47.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:57:47.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:57:47.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:47.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:48.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T13:57:48.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' -1 2024-03-09T13:57:48.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:57:48.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=3 2024-03-09T13:57:48.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:57:48.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:57:48.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:57:48.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:57:48.127 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:57:48.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:57:48.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:57:48.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:57:48.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:57:48.128 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:57:48.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:57:48.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:57:48.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:48.395 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:48.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T13:57:48.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:57:48.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:57:48.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:57:48.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:57:48.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:57:48.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:57:48.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:57:48.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:57:49.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:57:49.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:57:49.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:57:49.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:57:49.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:57:49.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:57:49.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:57:49.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:57:49.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:57:49.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:57:49.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:57:49.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:57:49.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:57:49.550 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:57:49.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:57:49.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:49.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:49.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T13:57:49.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:57:49.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:57:49.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:57:49.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:57:49.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:57:49.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:57:49.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:57:49.893 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:57:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:57:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:57:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-09T13:57:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:57:50.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-09T13:57:50.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:57:50.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:57:50.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:57:50.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:57:50.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:57:50.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:57:50.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:57:50.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:57:50.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-03-09T13:57:50.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:57:50.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:50.904 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:51.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 6 2024-03-09T13:57:51.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 3 '!=' 3 2024-03-09T13:57:51.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:57:51.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:57:51.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:57:51.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:57:51.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:57:51.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:57:51.240 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:57:51.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:57:51.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:57:51.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-03-09T13:57:51.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:57:51.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-03-09T13:57:52.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:57:52.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:57:52.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:57:52.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:57:52.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:57:52.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:57:52.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:57:52.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:57:52.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:57:52.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:57:52.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:57:52.691 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:57:53.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:57:53.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:57:53.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:57:53.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2107: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2109: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ11 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T13:57:53.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ11 2024-03-09T13:57:53.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T13:57:53.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ11 2024-03-09T13:57:53.041 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ11 2024-03-09T13:57:53.041 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T13:57:53.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T13:57:53.317 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T13:57:53.317 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T13:57:53.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T13:57:53.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T13:57:53.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-09T13:57:53.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T13:57:53.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T13:57:53.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-09T13:57:53.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T13:57:53.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-09T13:57:53.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T13:57:53.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-09T13:57:53.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T13:57:53.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T13:57:53.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T13:57:53.320 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-09T13:57:53.320 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T13:57:53.320 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T13:57:53.320 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:53.320 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:53.320 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:53.320 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.1.asok 2024-03-09T13:57:53.321 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70042/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-09T13:57:53.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T13:57:53.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T13:57:53.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T13:57:53.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-09T13:57:53.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T13:57:53.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T13:57:53.408 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:53.408 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:53.408 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:53.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.1.asok 2024-03-09T13:57:53.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T13:57:53.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70042/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2024-03-09T13:57:53.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2110: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2024-03-09T13:57:53.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T13:57:53.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T13:57:53.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-09T13:57:53.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T13:57:53.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T13:57:53.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T13:57:53.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ12 2024-03-09T13:57:53.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T13:57:53.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T13:57:53.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T13:57:53.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-09T13:57:53.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T13:57:53.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ12 2024-03-09T13:57:53.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T13:57:53.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ12 2024-03-09T13:57:53.497 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ12 2024-03-09T13:57:53.497 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T13:57:53.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T13:57:53.773 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T13:57:53.773 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T13:57:53.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T13:57:53.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T13:57:53.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-09T13:57:53.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T13:57:53.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T13:57:53.774 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-09T13:57:53.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T13:57:53.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-09T13:57:53.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T13:57:53.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-09T13:57:53.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T13:57:53.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T13:57:53.776 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T13:57:53.776 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-09T13:57:53.776 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T13:57:53.776 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T13:57:53.776 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:53.776 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:53.776 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:53.776 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.0.asok 2024-03-09T13:57:53.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70042/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-09T13:57:53.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T13:57:53.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T13:57:53.864 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T13:57:53.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-09T13:57:53.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T13:57:53.864 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T13:57:53.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:53.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:53.865 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:53.865 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.0.asok 2024-03-09T13:57:53.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T13:57:53.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70042/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2024-03-09T13:57:53.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2111: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2024-03-09T13:57:53.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T13:57:53.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-09T13:57:53.955 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-09T13:57:53.956 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T13:57:54.228 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T13:57:54.228 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T13:57:54.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T13:57:54.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T13:57:54.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T13:57:54.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-03-09T13:57:54.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T13:57:54.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T13:57:54.229 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/0/type 2024-03-09T13:57:54.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T13:57:54.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-03-09T13:57:54.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T13:57:54.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-03-09T13:57:54.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T13:57:54.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T13:57:54.231 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T13:57:54.231 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-03-09T13:57:54.231 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T13:57:54.231 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T13:57:54.231 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:54.232 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:54.232 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:54.232 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.0.asok 2024-03-09T13:57:54.232 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70042/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-03-09T13:57:54.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T13:57:54.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T13:57:54.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T13:57:54.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-03-09T13:57:54.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-03-09T13:57:54.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-03-09T13:57:54.319 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:54.319 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:54.319 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:54.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.0.asok 2024-03-09T13:57:54.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T13:57:54.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70042/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2024-03-09T13:57:54.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2112: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2024-03-09T13:57:54.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=rep 2024-03-09T13:57:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-03-09T13:57:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-03-09T13:57:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-03-09T13:57:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=csr_pool 2024-03-09T13:57:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-03-09T13:57:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=ROBJ13 2024-03-09T13:57:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-03-09T13:57:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/osd-scrub-repair 2024-03-09T13:57:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-03-09T13:57:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-03-09T13:57:54.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-03-09T13:57:54.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds csr_pool ROBJ13 2024-03-09T13:57:54.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=csr_pool 2024-03-09T13:57:54.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=ROBJ13 2024-03-09T13:57:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map csr_pool ROBJ13 2024-03-09T13:57:54.410 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T13:57:54.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-03-09T13:57:54.687 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T13:57:54.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-03-09T13:57:54.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('1' '0') 2024-03-09T13:57:54.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-03-09T13:57:54.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-03-09T13:57:54.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' rep '!=' ec ']' 2024-03-09T13:57:54.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: shard_id= 2024-03-09T13:57:54.687 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/osd-scrub-repair/1/type 2024-03-09T13:57:54.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-03-09T13:57:54.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-03-09T13:57:54.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-03-09T13:57:54.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-03-09T13:57:54.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-03-09T13:57:54.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-03-09T13:57:54.689 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-03-09T13:57:54.689 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-03-09T13:57:54.689 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T13:57:54.690 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T13:57:54.690 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:54.690 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:54.690 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:54.690 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.1.asok 2024-03-09T13:57:54.690 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.70042/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-03-09T13:57:54.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-03-09T13:57:54.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-03-09T13:57:54.777 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-03-09T13:57:54.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-03-09T13:57:54.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-03-09T13:57:54.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-03-09T13:57:54.777 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:57:54.777 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:57:54.778 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:57:54.778 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-osd.1.asok 2024-03-09T13:57:54.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-03-09T13:57:54.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.70042/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2024-03-09T13:57:54.865 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2115: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2024-03-09T13:57:54.961 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0: {} 2024-03-09T13:57:54.962 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0: osd_max_object_size = '' 2024-03-09T13:57:54.978 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1: {} 2024-03-09T13:57:54.979 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1: osd_max_object_size = '' 2024-03-09T13:57:54.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2117: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2024-03-09T13:57:54.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=3.0 2024-03-09T13:57:54.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:57:54.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:57:54.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:57:54.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:57:54.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:57:55.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-09T13:53:15.768738+0000 2024-03-09T13:57:55.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 3.0 2024-03-09T13:57:55.521 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2024-03-09T13:57:55.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 3.0 2024-03-09T13:53:15.768738+0000 last_deep_scrub_stamp 2024-03-09T13:57:55.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T13:57:55.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:53:15.768738+0000 2024-03-09T13:57:55.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-09T13:57:55.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:57:55.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:57:55.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:57:55.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:57:55.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:57:55.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:57:55.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:57:55.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:53:15.768738+0000 '>' 2024-03-09T13:53:15.768738+0000 2024-03-09T13:57:55.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:57:56.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:57:56.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:57:56.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:57:56.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:57:56.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:57:56.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:57:56.805 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:57:57.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:53:15.768738+0000 '>' 2024-03-09T13:53:15.768738+0000 2024-03-09T13:57:57.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:57:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:57:58.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:57:58.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:57:58.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:57:58.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:57:58.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:57:58.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:57:58.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:53:15.768738+0000 '>' 2024-03-09T13:53:15.768738+0000 2024-03-09T13:57:58.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:57:59.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:57:59.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:57:59.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:57:59.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:57:59.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:57:59.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:57:59.355 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:57:59.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:53:15.768738+0000 '>' 2024-03-09T13:53:15.768738+0000 2024-03-09T13:57:59.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:58:00.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:58:00.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:58:00.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2024-03-09T13:58:00.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:58:00.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T13:58:00.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:58:00.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2024-03-09T13:58:00.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:57:56.186896+0000 '>' 2024-03-09T13:53:15.768738+0000 2024-03-09T13:58:00.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:58:00.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2119: TEST_corrupt_scrub_replicated: err_strings=() 2024-03-09T13:58:00.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2120: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2024-03-09T13:58:00.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2121: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2024-03-09T13:58:00.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2122: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2024-03-09T13:58:00.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2123: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2024-03-09T13:58:00.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2124: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2024-03-09T13:58:00.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2125: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2024-03-09T13:58:00.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2126: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2024-03-09T13:58:00.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2127: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2024-03-09T13:58:00.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2128: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2024-03-09T13:58:00.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2129: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-03-09T13:58:00.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2130: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2024-03-09T13:58:00.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2131: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2024-03-09T13:58:00.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2132: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2024-03-09T13:58:00.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2133: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2024-03-09T13:58:00.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2134: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2024-03-09T13:58:00.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2135: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-03-09T13:58:00.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2136: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2024-03-09T13:58:00.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2137: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2024-03-09T13:58:00.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2138: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2024-03-09T13:58:00.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2139: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2024-03-09T13:58:00.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2140: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2024-03-09T13:58:00.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2141: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2024-03-09T13:58:00.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2142: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2024-03-09T13:58:00.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2143: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2024-03-09T13:58:00.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2144: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2024-03-09T13:58:00.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2145: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2024-03-09T13:58:00.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2146: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2024-03-09T13:58:00.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2147: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2024-03-09T13:58:00.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2148: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2024-03-09T13:58:00.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2149: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2024-03-09T13:58:00.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2150: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2024-03-09T13:58:00.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2151: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2024-03-09T13:58:00.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2152: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2024-03-09T13:58:00.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2153: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2024-03-09T13:58:00.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2154: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2024-03-09T13:58:00.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2155: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' 2024-03-09T13:58:00.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2156: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-09T13:58:00.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2157: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2024-03-09T13:58:00.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2158: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2024-03-09T13:58:00.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:00.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:00.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:00.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:00.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:00.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:00.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:00.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:00.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:00.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:00.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:00.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:00.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:00.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:00.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:00.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:00.981 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:00.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:00.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:00.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.069 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.079 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.152 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* no longer understand old encoding version 3 < 97: Malformed input' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2160: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2024-03-09T13:58:01.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2162: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2024-03-09T13:58:01.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2169: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2024-03-09T13:58:01.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2024-03-09T13:58:01.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2171: TEST_corrupt_scrub_replicated: test 1 = 1 2024-03-09T13:58:01.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T13:58:01.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2024-03-09T13:58:01.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2024-03-09T13:58:01.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2024-03-09T13:58:01.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2177: TEST_corrupt_scrub_replicated: epoch=211 2024-03-09T13:58:01.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-09T13:58:01.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2179: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T13:58:01.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2024-03-09T13:58:01.377 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T13:58:01.377 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T13:58:01.377 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T13:58:01.377 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T13:58:01.377 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T13:58:01.377 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T13:58:01.377 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T13:58:01.377 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T13:58:01.378 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T13:58:01.378 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T13:58:01.378 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T13:58:01.378 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T13:58:01.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: jq .inconsistents 2024-03-09T13:58:01.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3558: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T13:58:01.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3559: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T13:58:01.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T13:58:01.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3560: TEST_corrupt_scrub_replicated: test no = yes 2024-03-09T13:58:01.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3565: TEST_corrupt_scrub_replicated: test '' = yes 2024-03-09T13:58:01.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3570: TEST_corrupt_scrub_replicated: repair 3.0 2024-03-09T13:58:01.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=3.0 2024-03-09T13:58:01.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 3.0 2024-03-09T13:58:01.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:58:01.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:58:01.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:58:01.403 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:58:01.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T13:57:56.186896+0000 2024-03-09T13:58:01.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 3.0 2024-03-09T13:58:01.929 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 3.0 on osd.1 to repair 2024-03-09T13:58:01.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 3.0 2024-03-09T13:57:56.186896+0000 2024-03-09T13:58:01.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T13:58:01.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:57:56.186896+0000 2024-03-09T13:58:01.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T13:58:01.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T13:58:01.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:58:01.942 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:58:01.942 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:58:01.942 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:58:01.942 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:58:01.942 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:58:02.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:57:56.186896+0000 '>' 2024-03-09T13:57:56.186896+0000 2024-03-09T13:58:02.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:58:03.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:58:03.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:58:03.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:58:03.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:58:03.213 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:58:03.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:58:03.214 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:58:03.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:57:56.186896+0000 '>' 2024-03-09T13:57:56.186896+0000 2024-03-09T13:58:03.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:58:04.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:58:04.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:58:04.551 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:58:04.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:58:04.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:58:04.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:58:04.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:58:04.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:57:56.186896+0000 '>' 2024-03-09T13:57:56.186896+0000 2024-03-09T13:58:04.853 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:58:05.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:58:05.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:58:05.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:58:05.855 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:58:05.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:58:05.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:58:05.856 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:58:06.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:57:56.186896+0000 '>' 2024-03-09T13:57:56.186896+0000 2024-03-09T13:58:06.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T13:58:07.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T13:58:07.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T13:58:07.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T13:58:07.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T13:58:07.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T13:58:07.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T13:58:07.136 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T13:58:07.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:58:03.071034+0000 '>' 2024-03-09T13:57:56.186896+0000 2024-03-09T13:58:07.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T13:58:07.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3571: TEST_corrupt_scrub_replicated: wait_for_clean 2024-03-09T13:58:07.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:58:07.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:58:07.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:58:07.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:58:07.408 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:58:07.408 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:58:07.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:58:07.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:58:07.409 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:58:07.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:58:07.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:58:07.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:58:07.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:58:07.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:58:07.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:58:07.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:58:07.772 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:58:07.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:58:07.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:58:07.772 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:58:07.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=880468295690 2024-03-09T13:58:07.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 880468295690 2024-03-09T13:58:07.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295690' 2024-03-09T13:58:07.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:58:07.907 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:58:08.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=906238099463 2024-03-09T13:58:08.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 906238099463 2024-03-09T13:58:08.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-880468295690 1-906238099463' 2024-03-09T13:58:08.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:58:08.011 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-880468295690 2024-03-09T13:58:08.012 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:58:08.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:58:08.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-880468295690 2024-03-09T13:58:08.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:58:08.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=880468295690 2024-03-09T13:58:08.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 880468295690' 2024-03-09T13:58:08.015 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 880468295690 2024-03-09T13:58:08.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:58:08.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295689 -lt 880468295690 2024-03-09T13:58:08.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:58:09.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:58:09.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:58:09.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 880468295690 -lt 880468295690 2024-03-09T13:58:09.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:58:09.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-906238099463 2024-03-09T13:58:09.569 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:58:09.569 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:58:09.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-906238099463 2024-03-09T13:58:09.570 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:58:09.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=906238099463 2024-03-09T13:58:09.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 906238099463' 2024-03-09T13:58:09.572 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 906238099463 2024-03-09T13:58:09.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:58:09.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 906238099463 -lt 906238099463 2024-03-09T13:58:09.847 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:58:09.847 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:58:09.847 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:58:10.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:58:10.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:58:10.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:58:10.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:58:10.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:58:10.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:58:10.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:58:10.187 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:58:10.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:58:10.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:58:10.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:58:10.458 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:58:10.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:58:10.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:58:10.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:58:10.800 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3574: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-09T13:58:10.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3575: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-09T13:58:10.880 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3577: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-09T13:58:10.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3578: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2024-03-09T13:58:10.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3579: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-09T13:58:10.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3580: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2024-03-09T13:58:10.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3582: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2024-03-09T13:58:10.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3588: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-03-09T13:58:11.266 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'csr_pool' removed 2024-03-09T13:58:11.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T13:58:11.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:58:11.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:58:11.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:58:11.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:58:11.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:58:11.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:58:11.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:58:11.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:58:11.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:58:11.387 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:58:11.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:58:11.388 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:58:11.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:58:11.389 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:58:11.389 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:58:11.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:58:11.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:58:11.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:58:11.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:58:11.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:58:11.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:58:11.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:58:11.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:58:11.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:58:11.428 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:58:11.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:58:11.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:58:11.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:58:11.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T13:58:11.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T13:58:11.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T13:58:11.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T13:58:11.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T13:58:11.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T13:58:11.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T13:58:11.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:58:11.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:58:11.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:58:11.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:58:11.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:58:11.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:58:11.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T13:58:11.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T13:58:11.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T13:58:11.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T13:58:11.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T13:58:11.435 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T13:58:11.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:58:11.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T13:58:11.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T13:58:11.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T13:58:11.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T13:58:11.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T13:58:11.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T13:58:11.440 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T13:58:11.440 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:58:11.440 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:58:11.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T13:58:11.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T13:58:11.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T13:58:11.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T13:58:11.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T13:58:11.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:58:11.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:58:11.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T13:58:11.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T13:58:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T13:58:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T13:58:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T13:58:11.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2024-03-09T13:58:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5931: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2024-03-09T13:58:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5932: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2024-03-09T13:58:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5933: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2024-03-09T13:58:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5935: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-09T13:58:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T13:58:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T13:58:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T13:58:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T13:58:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T13:58:11.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-09T13:58:11.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T13:58:11.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:58:11.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:58:11.916 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:58:11.916 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:58:11.916 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:58:11.916 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:58:11.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-09T13:58:12.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T13:58:12.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T13:58:12.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:58:12.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:58:12.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T13:58:12.093 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T13:58:12.093 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:58:12.093 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:58:12.093 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:58:12.094 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:58:12.094 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:58:12.094 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:58:12.094 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:58:12.094 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:58:12.094 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T13:58:12.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T13:58:12.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T13:58:12.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T13:58:12.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T13:58:12.175 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T13:58:12.175 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T13:58:12.175 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T13:58:12.175 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T13:58:12.175 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T13:58:12.175 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:58:12.176 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:58:12.176 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T13:58:12.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T13:58:12.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T13:58:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5936: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2024-03-09T13:58:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T13:58:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T13:58:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T13:58:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T13:58:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T13:58:12.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T13:58:13.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T13:58:13.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:58:13.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:58:13.069 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:58:13.069 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:58:13.069 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:58:13.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:58:13.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:58:13.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T13:58:13.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5937: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2024-03-09T13:58:13.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:58:13.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:58:13.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T13:58:13.100 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:58:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:58:13.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:58:13.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:58:13.103 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:58:13.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c53864ff-2e80-4853-9022-44cdfb579a06 2024-03-09T13:58:13.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c53864ff-2e80-4853-9022-44cdfb579a06' 2024-03-09T13:58:13.103 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 c53864ff-2e80-4853-9022-44cdfb579a06 2024-03-09T13:58:13.104 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:58:13.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1auxlCH0aBxAA1sU4cNBqzrhtdeU77IalAA== 2024-03-09T13:58:13.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1auxlCH0aBxAA1sU4cNBqzrhtdeU77IalAA=="}' 2024-03-09T13:58:13.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c53864ff-2e80-4853-9022-44cdfb579a06 -i td/osd-scrub-repair/0/new.json 2024-03-09T13:58:13.325 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:58:13.335 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T13:58:13.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1auxlCH0aBxAA1sU4cNBqzrhtdeU77IalAA== --osd-uuid c53864ff-2e80-4853-9022-44cdfb579a06 2024-03-09T13:58:13.363 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:13.362+0000 7fa093a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:58:13.375 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:13.375+0000 7fa093a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:58:13.377 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:13.376+0000 7fa093a1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:58:13.377 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:13.376+0000 7fa093a1c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:58:13.377 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:13.376+0000 7fa093a1c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:58:13.377 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:13.376+0000 7fa093a1c640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T13:58:13.468 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:13.467+0000 7fa093a1c640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T13:58:19.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T13:58:19.166 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:58:19.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T13:58:19.167 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T13:58:19.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:58:19.523 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T13:58:19.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T13:58:19.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:58:19.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:58:19.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:58:19.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:58:19.549 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:19.548+0000 7f86f6cc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:58:19.551 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:19.550+0000 7f86f6cc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:58:19.552 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:19.551+0000 7f86f6cc8640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:58:19.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T13:58:19.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:58:19.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:58:19.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:58:19.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:58:19.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:19.802 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:58:19.802 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:58:19.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:19.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:20.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:20.085 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:20.085+0000 7f86f6cc8640 -1 Falling back to public interface 2024-03-09T13:58:21.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:21.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:21.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:58:21.074 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:58:21.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:21.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:21.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:22.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:22.349 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:22.350 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:58:22.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:58:22.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:22.350 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:22.408 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:22.407+0000 7f86f6cc8640 -1 osd.0 0 log_to_monitors true 2024-03-09T13:58:22.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:23.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:23.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:23.677 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:58:23.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:58:23.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:23.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:23.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:24.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:24.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:24.965 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:58:24.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:58:24.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:24.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:25.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:26.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:26.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:26.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:58:26.360 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:58:26.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:26.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:26.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:27.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:27.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:27.645 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:58:27.646 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:58:27.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:27.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:27.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:28.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:28.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:28.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:58:28.964 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:58:28.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:28.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:29.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:30.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:30.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:30.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:58:30.333 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:58:30.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:30.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:30.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:31.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:31.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:31.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:58:31.620 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:58:31.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:31.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:31.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:32.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:32.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:32.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:58:32.955 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:58:32.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:32.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:33.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:34.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:34.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:34.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:58:34.334 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:58:34.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:34.334 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:35.187 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:36.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:36.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:36.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:58:36.189 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:58:36.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:36.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:36.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:37.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:37.701 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:58:37.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:58:37.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:37.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:38.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:39.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:39.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:39.164 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:58:39.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:58:39.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:39.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:39.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:40.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:40.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:40.848 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:58:40.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:58:40.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:40.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:41.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:42.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:42.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:42.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:58:42.577 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:58:42.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:42.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:43.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:44.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:44.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:44.180 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:58:44.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:58:44.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:44.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:44.545 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:45.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:45.546 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:45.547 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:58:45.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:58:45.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:45.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:58:45.864 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/377884775,v1:127.0.0.1:6803/377884775] [v2:127.0.0.1:6804/377884775,v1:127.0.0.1:6805/377884775] exists,up c53864ff-2e80-4853-9022-44cdfb579a06 2024-03-09T13:58:45.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:58:45.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:58:45.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:58:45.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5938: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2024-03-09T13:58:45.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T13:58:45.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T13:58:45.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T13:58:45.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T13:58:45.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:58:45.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:58:45.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:58:45.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:58:45.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:58:45.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:58:45.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T13:58:45.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T13:58:45.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:58:45.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:58:45.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:58:45.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:58:45.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:58:45.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:58:45.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T13:58:45.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T13:58:45.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:58:45.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T13:58:45.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=27ad74e5-5a99-4c15-8a8a-cacb10f5da70 2024-03-09T13:58:45.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 27ad74e5-5a99-4c15-8a8a-cacb10f5da70' 2024-03-09T13:58:45.872 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 27ad74e5-5a99-4c15-8a8a-cacb10f5da70 2024-03-09T13:58:45.872 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T13:58:45.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVa+xlKHPQNBAAkwPKShZC58QNQ04gZgdLyA== 2024-03-09T13:58:45.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVa+xlKHPQNBAAkwPKShZC58QNQ04gZgdLyA=="}' 2024-03-09T13:58:45.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 27ad74e5-5a99-4c15-8a8a-cacb10f5da70 -i td/osd-scrub-repair/1/new.json 2024-03-09T13:58:46.224 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:58:46.235 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T13:58:46.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVa+xlKHPQNBAAkwPKShZC58QNQ04gZgdLyA== --osd-uuid 27ad74e5-5a99-4c15-8a8a-cacb10f5da70 2024-03-09T13:58:46.265 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:46.264+0000 7f0a53624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:58:46.310 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:46.308+0000 7f0a53624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:58:46.310 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:46.309+0000 7f0a53624640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:58:46.311 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:46.309+0000 7f0a53624640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:58:46.311 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:46.309+0000 7f0a53624640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:58:46.311 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:46.309+0000 7f0a53624640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T13:58:46.384 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:46.383+0000 7f0a53624640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T13:58:52.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T13:58:52.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T13:58:52.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T13:58:52.233 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T13:58:52.233 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T13:58:52.586 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T13:58:52.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T13:58:52.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:58:52.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T13:58:52.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T13:58:52.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T13:58:52.612 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:52.611+0000 7f706840e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:58:52.614 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:52.614+0000 7f706840e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:58:52.616 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:52.615+0000 7f706840e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:58:52.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T13:58:52.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:58:52.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:58:52.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:58:52.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:58:52.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:52.861 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T13:58:52.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:58:52.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:52.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:58:53.140 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:53.139+0000 7f706840e640 -1 Falling back to public interface 2024-03-09T13:58:53.151 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:54.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:54.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:54.153 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T13:58:54.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:58:54.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:54.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:58:54.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:55.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:55.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:55.429 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T13:58:55.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:58:55.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:55.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:58:55.665 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:58:55.664+0000 7f706840e640 -1 osd.1 0 log_to_monitors true 2024-03-09T13:58:55.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:56.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:56.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:56.708 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T13:58:56.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:58:56.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:56.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:58:56.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:57.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:57.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:57.982 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T13:58:57.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T13:58:57.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:57.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:58:58.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:58:59.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:58:59.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:58:59.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T13:58:59.277 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T13:58:59.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:58:59.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:58:59.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:00.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:00.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:00.810 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T13:59:00.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T13:59:00.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:00.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:01.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:02.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:02.093 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T13:59:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T13:59:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:02.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:02.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:03.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:03.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:03.498 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T13:59:03.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T13:59:03.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:03.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:03.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:04.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:04.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:04.836 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T13:59:04.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T13:59:04.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:04.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:05.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:06.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:06.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:06.121 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T13:59:06.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T13:59:06.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:06.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:06.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:07.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:07.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:07.478 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T13:59:07.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T13:59:07.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:07.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:07.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:08.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:08.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:08.769 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T13:59:08.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T13:59:08.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:08.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:09.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:10.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:10.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:10.287 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T13:59:10.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T13:59:10.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:10.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:10.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:11.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:11.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:11.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T13:59:11.881 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T13:59:11.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:11.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:12.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:13.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:13.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:13.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T13:59:13.281 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T13:59:13.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:13.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:14.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:15.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:15.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:15.093 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T13:59:15.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T13:59:15.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:15.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:15.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:16.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:16.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:16.670 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T13:59:16.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T13:59:16.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:16.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:17.188 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:18.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:18.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:18.190 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T13:59:18.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T13:59:18.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:18.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:18.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:19.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:19.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:19.692 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T13:59:19.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T13:59:19.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:19.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:19.970 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2867018688,v1:127.0.0.1:6811/2867018688] [v2:127.0.0.1:6812/2867018688,v1:127.0.0.1:6813/2867018688] exists,up 27ad74e5-5a99-4c15-8a8a-cacb10f5da70 2024-03-09T13:59:19.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:59:19.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:59:19.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:59:19.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5939: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2024-03-09T13:59:19.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T13:59:20.259 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T13:59:20.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T13:59:20.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T13:59:20.636 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T13:59:20.646 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:59:21.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T13:59:23.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5940: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-03-09T13:59:23.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:59:23.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:59:23.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:59:23.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:59:23.615 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:59:23.615 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:59:23.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:59:23.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:59:23.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:59:23.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:59:23.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:59:23.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:59:23.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:59:23.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:59:23.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:59:23.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:59:23.988 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:59:23.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:59:23.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:59:23.989 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:59:24.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836490 2024-03-09T13:59:24.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836490 2024-03-09T13:59:24.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490' 2024-03-09T13:59:24.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:59:24.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:59:24.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672963 2024-03-09T13:59:24.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672963 2024-03-09T13:59:24.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836490 1-42949672963' 2024-03-09T13:59:24.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:59:24.195 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836490 2024-03-09T13:59:24.196 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:59:24.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:59:24.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836490 2024-03-09T13:59:24.197 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:59:24.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836490 2024-03-09T13:59:24.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-03-09T13:59:24.199 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836490 2024-03-09T13:59:24.199 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:59:24.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836489 -lt 21474836490 2024-03-09T13:59:24.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:59:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:59:25.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:59:25.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836490 2024-03-09T13:59:25.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:59:25.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672963 2024-03-09T13:59:25.750 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:59:25.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:59:25.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672963 2024-03-09T13:59:25.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:59:25.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672963 2024-03-09T13:59:25.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2024-03-09T13:59:25.753 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672963 2024-03-09T13:59:25.753 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:59:26.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672963 -lt 42949672963 2024-03-09T13:59:26.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:59:26.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:59:26.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:59:26.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T13:59:26.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:59:26.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:59:26.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:59:26.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:59:26.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:59:26.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:59:26.367 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:59:26.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T13:59:26.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:59:26.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:59:26.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:59:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T13:59:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:59:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:59:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5942: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2024-03-09T13:59:26.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 1 2024-03-09T13:59:27.374 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'foo' created 2024-03-09T13:59:27.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:59:28.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5943: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2024-03-09T13:59:28.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create csr_pool 1 1 2024-03-09T13:59:28.833 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'csr_pool' created 2024-03-09T13:59:28.845 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T13:59:29.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5944: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2024-03-09T13:59:29.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:59:29.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:59:29.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:59:29.847 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:59:29.848 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:59:29.848 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:59:29.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:59:29.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:59:29.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:59:29.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:59:29.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:59:29.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:59:29.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:59:29.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:59:29.949 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:59:30.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:59:30.221 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:59:30.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:59:30.221 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:59:30.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:59:30.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836492 2024-03-09T13:59:30.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836492 2024-03-09T13:59:30.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492' 2024-03-09T13:59:30.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:59:30.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:59:30.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672965 2024-03-09T13:59:30.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672965 2024-03-09T13:59:30.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836492 1-42949672965' 2024-03-09T13:59:30.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:59:30.436 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836492 2024-03-09T13:59:30.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:59:30.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:59:30.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836492 2024-03-09T13:59:30.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:59:30.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836492 2024-03-09T13:59:30.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-03-09T13:59:30.440 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836492 2024-03-09T13:59:30.440 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:59:30.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836492 2024-03-09T13:59:30.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:59:31.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:59:31.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:59:31.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836493 -lt 21474836492 2024-03-09T13:59:31.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:59:31.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672965 2024-03-09T13:59:31.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:59:31.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:59:31.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672965 2024-03-09T13:59:31.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:59:32.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672965 2024-03-09T13:59:32.000 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672965 2024-03-09T13:59:32.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-03-09T13:59:32.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:59:32.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672965 -lt 42949672965 2024-03-09T13:59:32.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:59:32.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:59:32.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:59:32.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:59:32.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:59:32.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:59:32.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:59:32.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:59:32.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:59:32.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:59:32.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:59:32.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:59:32.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:59:32.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:59:32.883 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:59:33.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-09T13:59:33.231 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-03-09T13:59:33.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T13:59:33.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-03-09T13:59:33.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:59:33.232 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T13:59:33.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:59:33.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:59:33.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:59:33.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:59:33.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:59:33.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:59:33.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:59:33.333 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:59:33.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:59:33.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:59:33.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:59:33.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:59:33.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-09T13:59:33.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-03-09T13:59:33.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:59:33.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:59:33.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:59:33.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:59:33.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:59:33.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:59:33.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:59:34.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:59:34.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:59:34.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-03-09T13:59:34.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:59:34.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-03-09T13:59:34.508 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:59:34.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:59:34.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:59:34.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:59:34.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:59:34.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:59:34.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:59:34.509 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:59:34.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T13:59:34.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:59:34.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:59:34.780 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:59:35.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 6 2024-03-09T13:59:35.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-03-09T13:59:35.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-03-09T13:59:35.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-03-09T13:59:35.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-03-09T13:59:35.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-03-09T13:59:35.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-03-09T13:59:35.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-03-09T13:59:35.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-03-09T13:59:35.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-03-09T13:59:35.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-03-09T13:59:35.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-03-09T13:59:35.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T13:59:35.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-03-09T13:59:35.861 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T13:59:35.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:59:35.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:59:35.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:59:35.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:59:35.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:59:35.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:59:35.863 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:59:36.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:59:36.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:59:36.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:59:36.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:59:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:59:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:59:36.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:59:36.468 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-03-09T13:59:36.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-09T13:59:36.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-03-09T13:59:36.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2024-03-09T13:59:36.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:59:36.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:59:36.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2024-03-09T13:59:36.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:59:36.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:59:36.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:59:36.841 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:59:36.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:59:37.267 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:59:37.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:59:37.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:59:37.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:59:37.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2024-03-09T13:59:37.688 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2024-03-09T13:59:37.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5946: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-09T13:59:37.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5947: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-03-09T13:59:37.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5948: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2024-03-09T13:59:37.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T13:59:37.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2024-03-09T13:59:37.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2024-03-09T13:59:37.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T13:59:37.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T13:59:37.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T13:59:38.162 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T13:59:38.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T13:59:38.690 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T13:59:38.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T13:59:38.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T13:59:38.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2024-03-09T13:59:38.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5950: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2024-03-09T13:59:39.061 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5951: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2024-03-09T13:59:39.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2024-03-09T13:59:39.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=csr_pool 2024-03-09T13:59:39.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=ROBJ0 2024-03-09T13:59:39.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map csr_pool ROBJ0 2024-03-09T13:59:39.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T13:59:39.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5954: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2024-03-09T13:59:39.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2024-03-09T13:59:39.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=csr_pool 2024-03-09T13:59:39.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=ROBJ0 2024-03-09T13:59:39.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map csr_pool ROBJ0 2024-03-09T13:59:39.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T13:59:39.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5955: TEST_corrupt_snapset_scrub_rep: local primary=1 2024-03-09T13:59:39.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5957: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2024-03-09T13:59:39.937 INFO:tasks.workunit.client.0.smithi162.stdout:created pool csr_pool snap snap1 2024-03-09T13:59:39.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5958: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2024-03-09T13:59:39.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: seq 1 2 2024-03-09T13:59:39.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-09T13:59:39.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2024-03-09T13:59:39.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2024-03-09T13:59:39.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=1 2024-03-09T13:59:39.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-03-09T13:59:39.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5968: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2024-03-09T13:59:40.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5969: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-09T13:59:40.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:59:40.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:59:40.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-03-09T13:59:40.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:59:40.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-09T13:59:40.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:59:40.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:59:40.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-03-09T13:59:40.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:59:40.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T13:59:40.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:59:40.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:59:40.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:59:40.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:59:40.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:59:40.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:59:40.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2024-03-09T13:59:40.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:59:40.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:59:40.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T13:59:40.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:59:40.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:59:40.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2024-03-09T13:59:42.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2024-03-09T13:59:42.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:59:42.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:59:42.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T13:59:42.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:59:42.487 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:59:42.488 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:59:42.488 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:59:42.488 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:59:42.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:59:42.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:59:42.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:59:42.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:59:42.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:59:42.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:59:42.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:59:42.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:59:42.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:59:42.488 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:59:42.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T13:59:42.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T13:59:42.489 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T13:59:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:59:42.490 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T13:59:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T13:59:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:59:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:59:42.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:59:42.516 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:59:42.515+0000 7fccc5e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:59:42.518 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:59:42.517+0000 7fccc5e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:59:42.519 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:59:42.518+0000 7fccc5e24640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:59:42.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T13:59:42.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:59:42.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T13:59:42.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:59:42.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:59:42.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:42.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:59:42.773 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:59:42.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:42.773 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:43.035 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:59:43.035+0000 7fccc5e24640 -1 Falling back to public interface 2024-03-09T13:59:43.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:44.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:44.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:44.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:59:44.046 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:59:44.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:44.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:44.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:45.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:45.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:45.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:59:45.331 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:59:45.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:45.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:45.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:45.666 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:59:45.666+0000 7fccc5e24640 -1 osd.1 31 log_to_monitors true 2024-03-09T13:59:46.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:46.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:46.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:59:46.618 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:59:46.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:46.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T13:59:47.007 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/1735219839,v1:127.0.0.1:6811/1735219839] [v2:127.0.0.1:6812/1735219839,v1:127.0.0.1:6813/1735219839] exists,up 27ad74e5-5a99-4c15-8a8a-cacb10f5da70 2024-03-09T13:59:47.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:59:47.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:59:47.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:59:47.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:59:47.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:59:47.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:59:47.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:59:47.008 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:59:47.008 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:59:47.009 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:59:47.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:59:47.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:59:47.009 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:59:47.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:59:47.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:59:47.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:59:47.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:59:47.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:59:47.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:59:47.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:59:47.510 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:59:47.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:59:47.511 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:59:47.511 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:59:47.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836497 2024-03-09T13:59:47.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836497 2024-03-09T13:59:47.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497' 2024-03-09T13:59:47.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:59:47.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:59:47.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855362 2024-03-09T13:59:47.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855362 2024-03-09T13:59:47.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836497 1-150323855362' 2024-03-09T13:59:47.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:59:47.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836497 2024-03-09T13:59:47.718 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:59:47.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:59:47.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836497 2024-03-09T13:59:47.720 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:59:47.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836497 2024-03-09T13:59:47.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-03-09T13:59:47.721 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 21474836497 2024-03-09T13:59:47.722 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:59:47.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836497 2024-03-09T13:59:47.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:59:48.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:59:48.997 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:59:49.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836497 -lt 21474836497 2024-03-09T13:59:49.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:59:49.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855362 2024-03-09T13:59:49.287 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:59:49.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:59:49.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855362 2024-03-09T13:59:49.289 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:59:49.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855362 2024-03-09T13:59:49.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2024-03-09T13:59:49.290 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 150323855362 2024-03-09T13:59:49.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:59:49.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855362 -lt 150323855362 2024-03-09T13:59:49.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:59:49.566 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:59:49.567 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:59:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T13:59:49.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T13:59:49.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T13:59:49.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T13:59:49.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T13:59:49.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T13:59:49.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T13:59:49.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T13:59:50.176 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T13:59:50.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T13:59:50.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:59:50.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T13:59:50.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T13:59:50.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T13:59:50.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T13:59:50.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5960: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2024-03-09T13:59:50.517 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5961: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2024-03-09T13:59:50.517 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2024-03-09T13:59:50.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5964: TEST_corrupt_snapset_scrub_rep: local osd=0 2024-03-09T13:59:50.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5966: TEST_corrupt_snapset_scrub_rep: case $i in 2024-03-09T13:59:50.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5973: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2024-03-09T13:59:50.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5974: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-09T13:59:50.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T13:59:50.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T13:59:50.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T13:59:50.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T13:59:50.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-09T13:59:50.638 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T13:59:50.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T13:59:50.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T13:59:50.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T13:59:50.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T13:59:50.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T13:59:50.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T13:59:50.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T13:59:50.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T13:59:50.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T13:59:50.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T13:59:50.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2024-03-09T13:59:50.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T13:59:50.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T13:59:50.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T13:59:50.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T13:59:50.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:59:50.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2024-03-09T13:59:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2024-03-09T13:59:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T13:59:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T13:59:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T13:59:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T13:59:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T13:59:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T13:59:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T13:59:52.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:59:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T13:59:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T13:59:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T13:59:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T13:59:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T13:59:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T13:59:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T13:59:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T13:59:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T13:59:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T13:59:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T13:59:52.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T13:59:52.741 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T13:59:52.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T13:59:52.741 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T13:59:52.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T13:59:52.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T13:59:52.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T13:59:52.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T13:59:52.768 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:59:52.767+0000 7f80c0511640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:59:52.774 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:59:52.770+0000 7f80c0511640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:59:52.775 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:59:52.771+0000 7f80c0511640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T13:59:53.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T13:59:53.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T13:59:53.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T13:59:53.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T13:59:53.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T13:59:53.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:53.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T13:59:53.023 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T13:59:53.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:53.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:59:53.289 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:59:53.288+0000 7f80c0511640 -1 Falling back to public interface 2024-03-09T13:59:53.324 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:54.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:54.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:54.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T13:59:54.326 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T13:59:54.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:54.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:59:54.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:55.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:55.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:55.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T13:59:55.612 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T13:59:55.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:55.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:59:55.792 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T13:59:55.791+0000 7f80c0511640 -1 osd.0 37 log_to_monitors true 2024-03-09T13:59:55.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T13:59:56.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T13:59:56.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T13:59:56.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T13:59:56.901 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T13:59:56.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T13:59:56.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T13:59:57.218 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6802/1149571212,v1:127.0.0.1:6803/1149571212] [v2:127.0.0.1:6804/1149571212,v1:127.0.0.1:6805/1149571212] exists,up c53864ff-2e80-4853-9022-44cdfb579a06 2024-03-09T13:59:57.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T13:59:57.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T13:59:57.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T13:59:57.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T13:59:57.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T13:59:57.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T13:59:57.219 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T13:59:57.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T13:59:57.219 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T13:59:57.219 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T13:59:57.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T13:59:57.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T13:59:57.219 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T13:59:57.309 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T13:59:57.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T13:59:57.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T13:59:57.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T13:59:57.310 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T13:59:57.310 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T13:59:57.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T13:59:57.613 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T13:59:57.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T13:59:57.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:59:57.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T13:59:57.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659138 2024-03-09T13:59:57.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659138 2024-03-09T13:59:57.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659138' 2024-03-09T13:59:57.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T13:59:57.717 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T13:59:57.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855365 2024-03-09T13:59:57.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855365 2024-03-09T13:59:57.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659138 1-150323855365' 2024-03-09T13:59:57.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:59:57.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-176093659138 2024-03-09T13:59:57.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:59:57.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T13:59:57.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-176093659138 2024-03-09T13:59:57.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:59:57.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659138 2024-03-09T13:59:57.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 176093659138' 2024-03-09T13:59:57.824 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 176093659138 2024-03-09T13:59:57.824 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:59:58.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 176093659138 2024-03-09T13:59:58.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T13:59:59.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T13:59:59.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T13:59:59.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659138 -lt 176093659138 2024-03-09T13:59:59.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T13:59:59.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-150323855365 2024-03-09T13:59:59.406 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T13:59:59.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T13:59:59.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-150323855365 2024-03-09T13:59:59.408 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T13:59:59.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855365 2024-03-09T13:59:59.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 150323855365' 2024-03-09T13:59:59.409 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 150323855365 2024-03-09T13:59:59.410 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T13:59:59.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855365 -lt 150323855365 2024-03-09T13:59:59.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T13:59:59.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T13:59:59.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:00:00.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 6 == 0 2024-03-09T14:00:00.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:00:00.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:00:00.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:00:00.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:00:00.024 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:00:00.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:00:00.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:00:00.296 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-03-09T14:00:00.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:00:00.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:00:00.297 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:00:00.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 6 2024-03-09T14:00:00.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:00:00.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:00:00.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5979: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2024-03-09T14:00:00.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5981: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2024-03-09T14:00:00.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: local pgid=3.0 2024-03-09T14:00:00.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: get_last_scrub_stamp 3.0 2024-03-09T14:00:00.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T14:00:00.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:00:00.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:00:00.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T14:00:00.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1876: pg_scrub: local last_scrub=2024-03-09T13:59:28.714562+0000 2024-03-09T14:00:00.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1877: pg_scrub: ceph pg scrub 3.0 2024-03-09T14:00:01.166 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 3.0 on osd.1 to scrub 2024-03-09T14:00:01.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1878: pg_scrub: wait_for_scrub 3.0 2024-03-09T13:59:28.714562+0000 2024-03-09T14:00:01.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=3.0 2024-03-09T14:00:01.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T13:59:28.714562+0000 2024-03-09T14:00:01.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T14:00:01.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T14:00:01.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:00:01.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T14:00:01.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T14:00:01.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:00:01.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:00:01.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T14:00:01.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:59:28.714562+0000 '>' 2024-03-09T13:59:28.714562+0000 2024-03-09T14:00:01.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:00:02.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:00:02.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:00:02.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T14:00:02.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T14:00:02.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:00:02.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:00:02.465 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T14:00:02.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:59:28.714562+0000 '>' 2024-03-09T13:59:28.714562+0000 2024-03-09T14:00:02.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:00:03.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:00:03.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:00:03.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T14:00:03.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T14:00:03.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:00:03.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:00:03.748 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T14:00:04.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:59:28.714562+0000 '>' 2024-03-09T13:59:28.714562+0000 2024-03-09T14:00:04.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:00:05.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:00:05.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:00:05.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T14:00:05.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T14:00:05.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:00:05.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:00:05.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T14:00:05.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:59:28.714562+0000 '>' 2024-03-09T13:59:28.714562+0000 2024-03-09T14:00:05.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:00:06.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:00:06.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:00:06.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T14:00:06.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T14:00:06.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:00:06.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:00:06.319 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T14:00:06.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T13:59:28.714562+0000 '>' 2024-03-09T13:59:28.714562+0000 2024-03-09T14:00:06.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:00:07.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:00:07.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:00:07.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2024-03-09T14:00:07.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=3.0 2024-03-09T14:00:07.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:00:07.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:00:07.599 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2024-03-09T14:00:07.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:00:01.821167+0000 '>' 2024-03-09T13:59:28.714562+0000 2024-03-09T14:00:07.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T14:00:07.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5983: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2024-03-09T14:00:07.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2024-03-09T14:00:07.913 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5985: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2024-03-09T14:00:07.913 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2024-03-09T14:00:07.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5987: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2024-03-09T14:00:07.914 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5989: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2024-03-09T14:00:07.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-03-09T14:00:07.942 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T14:00:07.942 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T14:00:07.942 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T14:00:07.942 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T14:00:07.942 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T14:00:07.942 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T14:00:07.942 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T14:00:07.942 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T14:00:07.942 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T14:00:07.942 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T14:00:07.942 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T14:00:07.942 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2024-03-09T14:00:07.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-03-09T14:00:07.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5991: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T14:00:07.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2024-03-09T14:00:07.965 INFO:tasks.workunit.client.0.smithi162.stderr: . as $in 2024-03-09T14:00:07.965 INFO:tasks.workunit.client.0.smithi162.stderr: | if type == "object" then 2024-03-09T14:00:07.965 INFO:tasks.workunit.client.0.smithi162.stderr: reduce keys[] as $key 2024-03-09T14:00:07.965 INFO:tasks.workunit.client.0.smithi162.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2024-03-09T14:00:07.965 INFO:tasks.workunit.client.0.smithi162.stderr: elif type == "array" then map( walk(f) ) | f 2024-03-09T14:00:07.965 INFO:tasks.workunit.client.0.smithi162.stderr: else f 2024-03-09T14:00:07.965 INFO:tasks.workunit.client.0.smithi162.stderr: end; 2024-03-09T14:00:07.965 INFO:tasks.workunit.client.0.smithi162.stderr:walk(if type == "object" then del(.mtime) else . end) 2024-03-09T14:00:07.966 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2024-03-09T14:00:07.966 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2024-03-09T14:00:07.966 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.version) else . end) 2024-03-09T14:00:07.966 INFO:tasks.workunit.client.0.smithi162.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2024-03-09T14:00:07.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2024-03-09T14:00:07.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6157: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2024-03-09T14:00:07.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T14:00:07.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2024-03-09T14:00:07.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6159: TEST_corrupt_snapset_scrub_rep: test no = yes 2024-03-09T14:00:07.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6164: TEST_corrupt_snapset_scrub_rep: test '' = yes 2024-03-09T14:00:07.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6169: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2024-03-09T14:00:07.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6170: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2024-03-09T14:00:07.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6171: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2024-03-09T14:00:07.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6172: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2024-03-09T14:00:07.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6173: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2024-03-09T14:00:07.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6174: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2024-03-09T14:00:07.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6175: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2024-03-09T14:00:07.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6176: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2024-03-09T14:00:07.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-09T14:00:07.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-03-09T14:00:07.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-09T14:00:07.991 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2024-03-09T14:00:07.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-09T14:00:07.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2024-03-09T14:00:07.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-09T14:00:07.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2024-03-09T14:00:07.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-09T14:00:07.998 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2024-03-09T14:00:08.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6178: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2024-03-09T14:00:08.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6180: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2024-03-09T14:00:08.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6187: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2024-03-09T14:00:08.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6193: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2024-03-09T14:00:08.466 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'csr_pool' removed 2024-03-09T14:00:08.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T14:00:08.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T14:00:08.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T14:00:08.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T14:00:08.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:00:08.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:00:08.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:00:08.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:00:08.479 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:00:08.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:00:08.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T14:00:08.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T14:00:08.587 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T14:00:08.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T14:00:08.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T14:00:08.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T14:00:08.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:00:08.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T14:00:08.590 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T14:00:08.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:00:08.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T14:00:08.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T14:00:08.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T14:00:08.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T14:00:08.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:00:08.605 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:00:08.605 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T14:00:08.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T14:00:08.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T14:00:08.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T14:00:08.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T14:00:08.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T14:00:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T14:00:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T14:00:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T14:00:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T14:00:08.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:00:08.608 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:00:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:00:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:00:08.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:00:08.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:00:08.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T14:00:08.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T14:00:08.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T14:00:08.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T14:00:08.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T14:00:08.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T14:00:08.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:00:08.612 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T14:00:08.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T14:00:08.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:00:08.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T14:00:08.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T14:00:08.615 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T14:00:08.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T14:00:08.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:00:08.616 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:00:08.616 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T14:00:08.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T14:00:08.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T14:00:08.617 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T14:00:08.619 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T14:00:08.619 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:00:08.619 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:00:08.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T14:00:08.620 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T14:00:08.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T14:00:08.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T14:00:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T14:00:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2024-03-09T14:00:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:978: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2024-03-09T14:00:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T14:00:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=false 2024-03-09T14:00:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-03-09T14:00:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-09T14:00:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T14:00:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T14:00:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T14:00:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T14:00:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T14:00:08.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T14:00:09.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T14:00:09.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:00:09.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:00:09.083 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:00:09.083 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:00:09.083 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:00:09.083 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:00:09.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T14:00:09.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T14:00:09.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T14:00:09.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T14:00:09.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T14:00:09.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T14:00:09.268 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T14:00:09.268 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T14:00:09.268 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T14:00:09.268 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T14:00:09.269 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T14:00:09.269 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:00:09.269 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:00:09.269 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T14:00:09.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T14:00:09.270 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T14:00:09.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T14:00:09.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T14:00:09.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T14:00:09.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T14:00:09.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T14:00:09.351 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T14:00:09.351 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T14:00:09.351 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T14:00:09.351 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T14:00:09.352 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:00:09.352 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:00:09.352 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T14:00:09.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T14:00:09.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T14:00:09.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-09T14:00:09.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T14:00:09.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T14:00:09.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T14:00:09.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T14:00:09.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T14:00:09.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T14:00:10.243 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T14:00:10.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:00:10.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:00:10.244 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:00:10.244 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:00:10.244 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:00:10.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:00:10.244 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T14:00:10.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T14:00:10.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-03-09T14:00:10.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:00:10.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-03-09T14:00:10.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T14:00:10.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T14:00:10.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T14:00:10.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T14:00:10.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:00:10.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:00:10.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:00:10.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:00:10.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:00:10.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T14:00:10.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T14:00:10.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T14:00:10.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T14:00:10.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:00:10.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T14:00:10.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:00:10.279 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:00:10.279 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:00:10.279 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:00:10.280 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:00:10.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:00:10.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:00:10.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:00:10.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T14:00:10.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T14:00:10.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:00:10.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:00:10.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:00:10.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:00:10.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:00:10.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:00:10.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T14:00:10.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T14:00:10.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T14:00:10.281 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T14:00:10.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=acf1ccdd-6b9d-4723-9c63-9cc11dfb7208 2024-03-09T14:00:10.282 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 acf1ccdd-6b9d-4723-9c63-9cc11dfb7208 2024-03-09T14:00:10.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 acf1ccdd-6b9d-4723-9c63-9cc11dfb7208' 2024-03-09T14:00:10.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T14:00:10.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBqa+xlDI2qERAA28wg/KL6oPwOAhbjIfsxKQ== 2024-03-09T14:00:10.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBqa+xlDI2qERAA28wg/KL6oPwOAhbjIfsxKQ=="}' 2024-03-09T14:00:10.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new acf1ccdd-6b9d-4723-9c63-9cc11dfb7208 -i td/osd-scrub-repair/0/new.json 2024-03-09T14:00:10.499 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:00:10.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T14:00:10.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBqa+xlDI2qERAA28wg/KL6oPwOAhbjIfsxKQ== --osd-uuid acf1ccdd-6b9d-4723-9c63-9cc11dfb7208 2024-03-09T14:00:10.539 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:10.539+0000 7f203a338640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:00:10.550 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:10.550+0000 7f203a338640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:00:10.551 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:10.552+0000 7f203a338640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:00:10.552 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:10.552+0000 7f203a338640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:00:10.552 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:10.552+0000 7f203a338640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:00:10.552 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:10.552+0000 7f203a338640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:00:10.642 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:10.642+0000 7f203a338640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T14:00:16.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T14:00:16.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T14:00:16.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T14:00:16.200 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T14:00:16.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T14:00:16.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T14:00:16.556 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T14:00:16.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T14:00:16.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T14:00:16.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T14:00:16.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T14:00:16.582 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:16.583+0000 7f05ed61f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:00:16.584 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:16.585+0000 7f05ed61f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:00:16.585 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:16.586+0000 7f05ed61f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:00:16.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T14:00:16.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:00:16.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T14:00:16.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:00:16.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:00:16.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:16.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:00:16.831 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:00:16.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:16.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:17.102 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:17.102+0000 7f05ed61f640 -1 Falling back to public interface 2024-03-09T14:00:17.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:18.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:18.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:18.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:00:18.104 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:00:18.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:18.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:18.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:19.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:19.387 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:19.387 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:00:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:00:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:19.388 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:19.484 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:19.484+0000 7f05ed61f640 -1 osd.0 0 log_to_monitors true 2024-03-09T14:00:19.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:20.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:20.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:20.671 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:00:20.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:00:20.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:20.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:20.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:21.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:21.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:21.959 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T14:00:21.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T14:00:21.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:21.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:22.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:23.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:23.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:23.243 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T14:00:23.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T14:00:23.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:23.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:23.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:24.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:24.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:24.526 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T14:00:24.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T14:00:24.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:24.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:24.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:25.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:25.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:25.812 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T14:00:25.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T14:00:25.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:25.813 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:26.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:27.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:27.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:27.096 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T14:00:27.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T14:00:27.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:27.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:27.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:28.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:28.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:28.476 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T14:00:28.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T14:00:28.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:28.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:28.762 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:29.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:29.763 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:29.763 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T14:00:29.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T14:00:29.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:29.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:30.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:31.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:31.077 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:31.077 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T14:00:31.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T14:00:31.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:31.078 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:31.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:32.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:32.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:32.476 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T14:00:32.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T14:00:32.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:32.477 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:33.204 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:34.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:34.205 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:34.206 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T14:00:34.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T14:00:34.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:34.206 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:34.794 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:35.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:35.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:35.796 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T14:00:35.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T14:00:35.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:35.796 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:36.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:37.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:37.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:37.379 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T14:00:37.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T14:00:37.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:37.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:37.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:38.900 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:38.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:38.901 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T14:00:38.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T14:00:38.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:38.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:39.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:40.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:40.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:40.702 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T14:00:40.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T14:00:40.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:40.703 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:41.490 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:42.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:42.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:42.492 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T14:00:42.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T14:00:42.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:42.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:00:42.804 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2843018166,v1:127.0.0.1:6803/2843018166] [v2:127.0.0.1:6804/2843018166,v1:127.0.0.1:6805/2843018166] exists,up acf1ccdd-6b9d-4723-9c63-9cc11dfb7208 2024-03-09T14:00:42.808 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:00:42.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:00:42.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:00:42.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:00:42.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-03-09T14:00:42.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T14:00:42.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T14:00:42.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T14:00:42.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T14:00:42.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:00:42.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:00:42.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:00:42.809 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:00:42.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:00:42.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T14:00:42.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T14:00:42.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T14:00:42.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T14:00:42.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:00:42.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T14:00:42.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:00:42.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:00:42.810 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:00:42.810 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:00:42.810 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:00:42.810 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T14:00:42.811 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T14:00:42.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eb1a7319-eaa0-445d-8e58-8d69ddf1830b 2024-03-09T14:00:42.812 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 eb1a7319-eaa0-445d-8e58-8d69ddf1830b 2024-03-09T14:00:42.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 eb1a7319-eaa0-445d-8e58-8d69ddf1830b' 2024-03-09T14:00:42.812 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T14:00:42.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCKa+xlgTs+MRAAJIb0Ri0sVllTEy9upm03XQ== 2024-03-09T14:00:42.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCKa+xlgTs+MRAAJIb0Ri0sVllTEy9upm03XQ=="}' 2024-03-09T14:00:42.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eb1a7319-eaa0-445d-8e58-8d69ddf1830b -i td/osd-scrub-repair/1/new.json 2024-03-09T14:00:43.167 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:00:43.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T14:00:43.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCKa+xlgTs+MRAAJIb0Ri0sVllTEy9upm03XQ== --osd-uuid eb1a7319-eaa0-445d-8e58-8d69ddf1830b 2024-03-09T14:00:43.206 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:43.206+0000 7ff56d6c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:00:43.243 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:43.242+0000 7ff56d6c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:00:43.244 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:43.243+0000 7ff56d6c4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:00:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:43.244+0000 7ff56d6c4640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:00:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:43.244+0000 7ff56d6c4640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:00:43.245 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:43.244+0000 7ff56d6c4640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:00:43.326 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:43.326+0000 7ff56d6c4640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T14:00:49.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T14:00:49.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T14:00:49.117 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T14:00:49.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T14:00:49.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T14:00:49.475 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T14:00:49.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T14:00:49.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T14:00:49.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T14:00:49.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T14:00:49.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T14:00:49.501 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:49.501+0000 7f43d0425640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:00:49.503 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:49.503+0000 7f43d0425640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:00:49.504 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:49.504+0000 7f43d0425640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:00:49.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T14:00:49.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:00:49.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T14:00:49.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:00:49.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:00:49.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:49.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:00:49.751 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:00:49.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:49.751 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:00:50.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:50.027 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:50.028+0000 7f43d0425640 -1 Falling back to public interface 2024-03-09T14:00:51.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:51.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:51.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:00:51.025 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:00:51.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:51.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:00:51.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:52.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:52.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:52.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:00:52.299 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:00:52.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:52.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:00:52.447 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:00:52.448+0000 7f43d0425640 -1 osd.1 0 log_to_monitors true 2024-03-09T14:00:52.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:53.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:53.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:53.693 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:00:53.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:00:53.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:53.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:00:53.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:54.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:54.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:54.978 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T14:00:54.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T14:00:54.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:54.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:00:55.312 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:56.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:56.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:56.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T14:00:56.314 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T14:00:56.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:56.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:00:56.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:57.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:57.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:57.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T14:00:57.681 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T14:00:57.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:57.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:00:57.961 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:00:58.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:00:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:00:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T14:00:58.963 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T14:00:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:00:58.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:00:59.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:00.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:00.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:00.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T14:01:00.279 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T14:01:00.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:00.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:01:00.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:01.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:01.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:01.564 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T14:01:01.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T14:01:01.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:01.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:01:01.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:02.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:02.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:02.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T14:01:02.850 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T14:01:02.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:02.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:01:03.138 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:04.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:04.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:04.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T14:01:04.140 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T14:01:04.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:04.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:01:04.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:05.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:05.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:05.430 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T14:01:05.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T14:01:05.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:05.430 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:01:05.930 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:06.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:06.931 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:06.932 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T14:01:06.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T14:01:06.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:06.932 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:01:07.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:08.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:08.518 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:08.519 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T14:01:08.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T14:01:08.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:08.519 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:01:09.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:10.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:10.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:10.359 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T14:01:10.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T14:01:10.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:10.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:01:11.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:12.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:12.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:12.286 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T14:01:12.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T14:01:12.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:12.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:01:13.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:14.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:14.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:14.155 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T14:01:14.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T14:01:14.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:14.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:01:14.731 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:15.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:15.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:15.733 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T14:01:15.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T14:01:15.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:15.733 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:01:16.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:17.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:17.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:17.021 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T14:01:17.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T14:01:17.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:17.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:01:17.299 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/251566529,v1:127.0.0.1:6811/251566529] [v2:127.0.0.1:6812/251566529,v1:127.0.0.1:6813/251566529] exists,up eb1a7319-eaa0-445d-8e58-8d69ddf1830b 2024-03-09T14:01:17.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:01:17.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:01:17.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:01:17.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:01:17.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T14:01:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:01:17.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:01:17.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:01:17.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:01:17.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:01:17.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T14:01:17.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T14:01:17.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T14:01:17.306 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T14:01:17.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d92569d8-0ec0-4131-9e80-1b3f883ebb38 2024-03-09T14:01:17.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d92569d8-0ec0-4131-9e80-1b3f883ebb38' 2024-03-09T14:01:17.306 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 d92569d8-0ec0-4131-9e80-1b3f883ebb38 2024-03-09T14:01:17.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T14:01:17.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCta+xlXQMiExAASndkBQVRRXNsMarP9mFIqw== 2024-03-09T14:01:17.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCta+xlXQMiExAASndkBQVRRXNsMarP9mFIqw=="}' 2024-03-09T14:01:17.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d92569d8-0ec0-4131-9e80-1b3f883ebb38 -i td/osd-scrub-repair/2/new.json 2024-03-09T14:01:17.685 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:01:17.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T14:01:17.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCta+xlXQMiExAASndkBQVRRXNsMarP9mFIqw== --osd-uuid d92569d8-0ec0-4131-9e80-1b3f883ebb38 2024-03-09T14:01:17.725 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:01:17.724+0000 7f80b800d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:01:17.761 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:01:17.761+0000 7f80b800d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:01:17.762 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:01:17.762+0000 7f80b800d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:01:17.763 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:01:17.762+0000 7f80b800d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T14:01:17.763 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:01:17.762+0000 7f80b800d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T14:01:17.763 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:01:17.762+0000 7f80b800d640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T14:01:17.836 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:01:17.835+0000 7f80b800d640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T14:01:24.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T14:01:24.055 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T14:01:24.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T14:01:24.056 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T14:01:24.056 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T14:01:24.414 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T14:01:24.414 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T14:01:24.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T14:01:24.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T14:01:24.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T14:01:24.415 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T14:01:24.441 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:01:24.440+0000 7f4e6280e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:01:24.451 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:01:24.450+0000 7f4e6280e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:01:24.452 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:01:24.451+0000 7f4e6280e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:01:24.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T14:01:24.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:01:24.692 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T14:01:24.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:01:24.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:01:24.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:24.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:01:24.693 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:01:24.693 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:24.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:24.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:24.981 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:01:24.981+0000 7f4e6280e640 -1 Falling back to public interface 2024-03-09T14:01:25.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:25.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:25.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:01:25.969 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:01:25.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:25.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:26.249 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:27.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:27.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:27.251 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:01:27.252 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:01:27.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:27.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:27.301 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:01:27.300+0000 7f4e6280e640 -1 osd.2 0 log_to_monitors true 2024-03-09T14:01:27.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:28.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:28.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:28.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:01:28.539 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:01:28.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:28.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:28.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:29.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:29.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:29.850 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T14:01:29.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T14:01:29.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:29.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:30.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:31.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:31.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:31.155 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T14:01:31.155 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T14:01:31.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:31.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:31.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:32.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:32.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:32.440 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T14:01:32.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T14:01:32.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:32.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:32.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:33.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:33.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:33.729 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T14:01:33.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T14:01:33.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:33.730 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:34.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:35.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:35.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:35.012 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T14:01:35.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T14:01:35.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:35.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:35.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:36.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:36.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:36.398 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T14:01:36.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T14:01:36.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:36.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:36.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:37.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:37.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:37.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T14:01:37.686 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T14:01:37.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:37.687 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:37.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:38.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:38.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:38.973 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T14:01:38.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T14:01:38.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:38.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:39.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:40.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:40.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:40.259 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T14:01:40.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T14:01:40.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:40.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:41.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:42.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:42.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:42.049 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T14:01:42.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T14:01:42.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:42.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:42.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:43.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:43.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:43.591 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T14:01:43.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T14:01:43.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:43.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:44.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:45.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:45.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:45.172 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T14:01:45.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T14:01:45.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:45.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:45.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:46.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:46.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:46.781 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T14:01:46.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T14:01:46.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:46.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:47.194 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:48.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:48.196 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:48.196 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T14:01:48.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T14:01:48.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:48.197 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:49.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:50.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:50.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:50.103 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T14:01:50.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T14:01:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:50.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:50.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:01:51.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:01:51.427 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:01:51.427 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T14:01:51.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T14:01:51.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:01:51.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:01:51.718 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/248010936,v1:127.0.0.1:6819/248010936] [v2:127.0.0.1:6820/248010936,v1:127.0.0.1:6821/248010936] exists,up d92569d8-0ec0-4131-9e80-1b3f883ebb38 2024-03-09T14:01:51.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:01:51.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:01:51.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:01:51.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-03-09T14:01:51.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T14:01:51.981 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T14:01:51.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T14:01:51.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T14:01:52.382 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T14:01:52.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T14:01:53.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T14:01:55.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-03-09T14:01:55.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:01:55.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:01:55.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:01:55.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:01:55.484 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:01:55.484 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:01:55.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:01:55.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:01:55.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:01:55.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:01:55.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:01:55.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:01:55.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:01:55.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:01:55.586 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:01:55.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:01:55.858 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:01:55.858 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T14:01:55.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:01:55.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:01:55.859 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:01:55.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836498 2024-03-09T14:01:55.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836498 2024-03-09T14:01:55.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498' 2024-03-09T14:01:55.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:01:55.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:01:56.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672975 2024-03-09T14:01:56.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672975 2024-03-09T14:01:56.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975' 2024-03-09T14:01:56.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:01:56.067 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T14:01:56.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-03-09T14:01:56.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-03-09T14:01:56.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836498 1-42949672975 2-64424509444' 2024-03-09T14:01:56.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:01:56.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836498 2024-03-09T14:01:56.171 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:01:56.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:01:56.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836498 2024-03-09T14:01:56.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:01:56.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836498 2024-03-09T14:01:56.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-03-09T14:01:56.174 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836498 2024-03-09T14:01:56.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:01:56.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836498 -lt 21474836498 2024-03-09T14:01:56.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:01:56.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672975 2024-03-09T14:01:56.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:01:56.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:01:56.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672975 2024-03-09T14:01:56.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:01:56.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672975 2024-03-09T14:01:56.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-03-09T14:01:56.473 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672975 2024-03-09T14:01:56.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:01:56.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672975 2024-03-09T14:01:56.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:01:56.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-03-09T14:01:56.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:01:56.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T14:01:56.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-03-09T14:01:56.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:01:56.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-03-09T14:01:56.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-03-09T14:01:56.750 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509444 2024-03-09T14:01:56.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T14:01:57.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509444 2024-03-09T14:01:57.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:01:57.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:01:57.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:01:57.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T14:01:57.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:01:57.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:01:57.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:01:57.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:01:57.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:01:57.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:01:57.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:01:57.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T14:01:57.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:01:57.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:01:57.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:01:57.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T14:01:57.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:01:57.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:01:57.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2024-03-09T14:01:57.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T14:01:57.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T14:01:57.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-09T14:01:57.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T14:01:57.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-09T14:01:58.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T14:01:58.405 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T14:01:59.054 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' created 2024-03-09T14:01:59.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T14:02:00.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-09T14:02:00.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T14:02:00.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:02:00.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:02:00.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:02:00.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:02:00.069 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:02:00.069 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:02:00.069 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:02:00.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:02:00.070 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:02:00.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:02:00.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:02:00.171 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:02:00.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:02:00.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:02:00.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:02:00.460 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:02:00.460 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:02:00.461 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T14:02:00.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:02:00.461 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:02:00.461 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:02:00.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-03-09T14:02:00.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-03-09T14:02:00.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-03-09T14:02:00.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:02:00.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:02:00.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-03-09T14:02:00.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-03-09T14:02:00.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672977' 2024-03-09T14:02:00.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:02:00.685 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T14:02:00.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509445 2024-03-09T14:02:00.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509445 2024-03-09T14:02:00.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509445' 2024-03-09T14:02:00.789 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:02:00.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-03-09T14:02:00.789 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:02:00.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:02:00.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-03-09T14:02:00.791 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:02:00.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-03-09T14:02:00.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-03-09T14:02:00.792 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836500 2024-03-09T14:02:00.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:02:01.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-09T14:02:01.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:02:02.063 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:02:02.064 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:02:02.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-09T14:02:02.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:02:03.347 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T14:02:03.348 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:02:03.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-03-09T14:02:03.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:02:03.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-03-09T14:02:03.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:02:03.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:02:03.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-03-09T14:02:03.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:02:03.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-03-09T14:02:03.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-03-09T14:02:03.635 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672977 2024-03-09T14:02:03.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:02:03.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672978 -lt 42949672977 2024-03-09T14:02:03.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:02:03.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509445 2024-03-09T14:02:03.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:02:03.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T14:02:03.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509445 2024-03-09T14:02:03.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:02:03.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509445 2024-03-09T14:02:03.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-03-09T14:02:03.912 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509445 2024-03-09T14:02:03.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T14:02:04.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509445 2024-03-09T14:02:04.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:02:04.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:02:04.186 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:02:04.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T14:02:04.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:02:04.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:02:04.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:02:04.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:02:04.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:02:04.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:02:04.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:02:04.812 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T14:02:04.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:02:04.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:02:04.813 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:02:05.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T14:02:05.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:02:05.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:02:05.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T14:02:05.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-03-09T14:02:05.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T14:02:05.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T14:02:05.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-03-09T14:02:05.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T14:02:05.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T14:02:05.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T14:02:05.674 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T14:02:05.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T14:02:06.202 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T14:02:06.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T14:02:06.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T14:02:06.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:02:06.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-03-09T14:02:06.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T14:02:06.524 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-03-09T14:02:06.524 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-03-09T14:02:06.525 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T14:02:06.818 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-09T14:02:06.819 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:02:06.819 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T14:02:06.819 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-09T14:02:06.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-03-09T14:02:06.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-03-09T14:02:06.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-03-09T14:02:06.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T14:02:06.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T14:02:06.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-03-09T14:02:06.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T14:02:06.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T14:02:06.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T14:02:07.190 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T14:02:07.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T14:02:07.592 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T14:02:07.603 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T14:02:07.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T14:02:07.604 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:02:07.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-03-09T14:02:07.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T14:02:07.956 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-03-09T14:02:07.956 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-03-09T14:02:07.957 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T14:02:08.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-09T14:02:08.325 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:02:08.325 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T14:02:08.325 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-09T14:02:08.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-03-09T14:02:08.325 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-03-09T14:02:08.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-03-09T14:02:08.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:02:08.326 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T14:02:08.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:02:08.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:02:08.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:02:08.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:02:08.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:02:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:02:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:02:08.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T14:02:08.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:02:08.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:02:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:02:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:02:08.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:02:09.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:02:09.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:02:09.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T14:02:09.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:02:09.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:02:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:02:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:02:09.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:02:10.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:02:10.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-03-09T14:02:10.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-03-09T14:02:11.337 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-03-09T14:02:11.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-03-09T14:02:11.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-03-09T14:02:13.075 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-03-09T14:02:13.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-03-09T14:02:13.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-03-09T14:02:14.368 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-03-09T14:02:14.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-03-09T14:02:14.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-03-09T14:02:16.041 INFO:tasks.workunit.client.0.smithi162.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-03-09T14:02:16.557 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-03-09T14:02:16.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:02:16.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-03-09T14:02:16.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T14:02:16.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T14:02:16.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T14:02:16.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T14:02:16.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:02:16.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:02:16.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:02:16.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:02:16.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:02:16.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T14:02:16.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T14:02:16.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T14:02:16.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T14:02:16.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:02:16.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T14:02:16.562 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T14:02:16.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T14:02:16.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T14:02:16.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T14:02:16.564 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T14:02:16.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T14:02:16.565 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T14:02:16.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T14:02:16.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T14:02:16.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T14:02:16.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T14:02:16.590 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:16.590+0000 7f24ebb9e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:16.593 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:16.592+0000 7f24ebb9e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:16.594 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:16.593+0000 7f24ebb9e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:16.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T14:02:16.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:02:16.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T14:02:16.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:02:16.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:02:16.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:16.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:02:16.848 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T14:02:16.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:16.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:02:17.107 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:17.106+0000 7f24ebb9e640 -1 Falling back to public interface 2024-03-09T14:02:17.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:02:18.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:02:18.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:18.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:02:18.122 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:02:18.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:18.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:02:18.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:02:19.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:02:19.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:19.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:02:19.408 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T14:02:19.408 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:19.409 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:02:19.506 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:19.505+0000 7f24ebb9e640 -1 osd.0 34 log_to_monitors true 2024-03-09T14:02:19.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:02:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:02:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:02:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T14:02:20.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:20.729 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:02:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 40 up_thru 40 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6802/3608677398,v1:127.0.0.1:6803/3608677398] [v2:127.0.0.1:6804/3608677398,v1:127.0.0.1:6805/3608677398] exists,up acf1ccdd-6b9d-4723-9c63-9cc11dfb7208 2024-03-09T14:02:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:02:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:02:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:02:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:02:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-03-09T14:02:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T14:02:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T14:02:21.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T14:02:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T14:02:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:02:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:02:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:02:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:02:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:02:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T14:02:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T14:02:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T14:02:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T14:02:21.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:02:21.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T14:02:21.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:02:21.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:02:21.015 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:02:21.015 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:02:21.015 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:02:21.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:02:21.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:02:21.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:02:21.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:02:21.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:02:21.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:02:21.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:02:21.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:02:21.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:02:21.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T14:02:21.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T14:02:21.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T14:02:21.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T14:02:21.016 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T14:02:21.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T14:02:21.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T14:02:21.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T14:02:21.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T14:02:21.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T14:02:21.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T14:02:21.034 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:21.033+0000 7f6154e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:21.036 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:21.035+0000 7f6154e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:21.037 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:21.036+0000 7f6154e12640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:21.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T14:02:21.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:02:21.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T14:02:21.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:02:21.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:02:21.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:21.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:02:21.284 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T14:02:21.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:21.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:02:21.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:02:21.556 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:21.555+0000 7f6154e12640 -1 Falling back to public interface 2024-03-09T14:02:22.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:02:22.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:22.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:02:22.557 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:02:22.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:22.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:02:22.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:02:23.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:02:23.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:23.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:02:23.834 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T14:02:23.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:23.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:02:23.867 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:23.867+0000 7f6154e12640 -1 osd.1 35 log_to_monitors true 2024-03-09T14:02:24.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:02:25.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:02:25.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:25.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:02:25.180 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T14:02:25.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:25.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:02:25.468 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/1505014285,v1:127.0.0.1:6811/1505014285] [v2:127.0.0.1:6812/1505014285,v1:127.0.0.1:6813/1505014285] exists,up eb1a7319-eaa0-445d-8e58-8d69ddf1830b 2024-03-09T14:02:25.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:02:25.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:02:25.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:02:25.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:02:25.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-03-09T14:02:25.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T14:02:25.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T14:02:25.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T14:02:25.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T14:02:25.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T14:02:25.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:02:25.469 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T14:02:25.470 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-03-09T14:02:25.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T14:02:25.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T14:02:25.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T14:02:25.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T14:02:25.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T14:02:25.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T14:02:25.494 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:25.495+0000 7fd60fc48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:25.496 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:25.497+0000 7fd60fc48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:25.497 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:25.498+0000 7fd60fc48640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:25.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T14:02:25.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:02:25.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T14:02:25.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:02:25.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:02:25.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:25.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:02:25.749 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T14:02:25.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:25.749 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:02:26.012 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:26.012+0000 7fd60fc48640 -1 Falling back to public interface 2024-03-09T14:02:26.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:02:27.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:02:27.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:27.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:02:27.023 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:02:27.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:27.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:02:27.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:02:28.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:02:28.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:28.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:02:28.307 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T14:02:28.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:28.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:02:28.471 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:28.455+0000 7fd60fc48640 -1 osd.2 37 log_to_monitors true 2024-03-09T14:02:28.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:02:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:02:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:02:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T14:02:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:29.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:02:29.912 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 38 last_clean_interval [15,37) [v2:127.0.0.1:6818/195892822,v1:127.0.0.1:6819/195892822] [v2:127.0.0.1:6820/195892822,v1:127.0.0.1:6821/195892822] exists,up d92569d8-0ec0-4131-9e80-1b3f883ebb38 2024-03-09T14:02:29.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:02:29.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:02:29.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:02:29.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-03-09T14:02:29.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T14:02:30.477 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' removed 2024-03-09T14:02:30.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T14:02:30.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T14:02:30.946 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T14:02:30.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T14:02:31.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T14:02:33.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-03-09T14:02:33.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:02:33.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:02:33.897 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:02:33.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:02:33.898 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:02:33.898 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:02:33.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:02:33.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:02:33.898 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:02:34.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:02:34.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:02:34.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:02:34.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:02:34.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:02:34.001 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:02:34.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:02:34.278 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:02:34.278 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T14:02:34.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:02:34.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:02:34.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:02:34.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691844 2024-03-09T14:02:34.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691844 2024-03-09T14:02:34.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844' 2024-03-09T14:02:34.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:02:34.420 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:02:34.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593732 2024-03-09T14:02:34.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593732 2024-03-09T14:02:34.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844 1-184683593732' 2024-03-09T14:02:34.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:02:34.525 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T14:02:34.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-03-09T14:02:34.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-03-09T14:02:34.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-171798691844 1-184683593732 2-201863462915' 2024-03-09T14:02:34.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:02:34.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-171798691844 2024-03-09T14:02:34.630 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:02:34.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:02:34.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-171798691844 2024-03-09T14:02:34.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:02:34.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691844 2024-03-09T14:02:34.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 171798691844' 2024-03-09T14:02:34.633 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 171798691844 2024-03-09T14:02:34.633 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:02:34.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691844 2024-03-09T14:02:34.906 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:02:35.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:02:35.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:02:36.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691843 -lt 171798691844 2024-03-09T14:02:36.189 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:02:37.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T14:02:37.191 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:02:37.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691845 -lt 171798691844 2024-03-09T14:02:37.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:02:37.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593732 2024-03-09T14:02:37.477 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:02:37.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:02:37.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593732 2024-03-09T14:02:37.479 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:02:37.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593732 2024-03-09T14:02:37.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593732' 2024-03-09T14:02:37.480 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 184683593732 2024-03-09T14:02:37.481 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:02:37.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593732 -lt 184683593732 2024-03-09T14:02:37.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:02:37.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-03-09T14:02:37.752 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:02:37.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T14:02:37.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-03-09T14:02:37.754 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:02:37.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-03-09T14:02:37.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-03-09T14:02:37.756 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 201863462915 2024-03-09T14:02:37.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T14:02:38.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-03-09T14:02:38.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:02:38.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:02:38.029 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:02:38.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T14:02:38.370 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:02:38.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:02:38.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:02:38.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:02:38.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:02:38.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:02:38.371 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:02:38.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T14:02:38.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:02:38.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:02:38.657 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:02:38.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T14:02:38.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:02:38.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:02:38.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-03-09T14:02:38.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T14:02:38.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-03-09T14:02:38.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-03-09T14:02:38.996 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T14:02:39.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-03-09T14:02:39.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-03-09T14:02:39.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T14:02:39.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T14:02:39.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:02:39.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:02:39.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:02:39.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:02:39.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T14:01:58.945757+0000 2024-03-09T14:02:39.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T14:02:39.801 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-03-09T14:02:39.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T14:01:58.945757+0000 2024-03-09T14:02:39.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T14:02:39.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T14:01:58.945757+0000 2024-03-09T14:02:39.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T14:02:39.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T14:02:39.814 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:02:39.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T14:02:39.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:02:39.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:02:39.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:02:39.814 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:02:40.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:01:58.945757+0000 '>' 2024-03-09T14:01:58.945757+0000 2024-03-09T14:02:40.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:02:41.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:02:41.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:02:41.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T14:02:41.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:02:41.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:02:41.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:02:41.086 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:02:41.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:01:58.945757+0000 '>' 2024-03-09T14:01:58.945757+0000 2024-03-09T14:02:41.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:02:42.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:02:42.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:02:42.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T14:02:42.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:02:42.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:02:42.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:02:42.368 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:02:42.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:01:58.945757+0000 '>' 2024-03-09T14:01:58.945757+0000 2024-03-09T14:02:42.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:02:43.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:02:43.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:02:43.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T14:02:43.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:02:43.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:02:43.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:02:43.664 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:02:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:01:58.945757+0000 '>' 2024-03-09T14:01:58.945757+0000 2024-03-09T14:02:43.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:02:44.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:02:44.942 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:02:44.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T14:02:44.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:02:44.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:02:44.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:02:44.943 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:02:45.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:02:40.402670+0000 '>' 2024-03-09T14:01:58.945757+0000 2024-03-09T14:02:45.222 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T14:02:45.223 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-03-09T14:02:45.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-03-09T14:02:45.223 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-03-09T14:02:45.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-03-09T14:02:45.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-03-09T14:02:45.224 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-03-09T14:02:45.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-03-09T14:02:45.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-03-09T14:02:45.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-03-09T14:02:45.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T14:02:45.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T14:02:45.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T14:02:45.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T14:02:45.332 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:02:45.332 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:02:45.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:02:45.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:02:45.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:02:45.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:02:45.441 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T14:02:45.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T14:02:45.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T14:02:45.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T14:02:45.443 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T14:02:45.444 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T14:02:45.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:02:45.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T14:02:45.445 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T14:02:45.445 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:02:45.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T14:02:45.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T14:02:45.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T14:02:45.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T14:02:45.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:02:45.464 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:02:45.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T14:02:45.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T14:02:45.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T14:02:45.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T14:02:45.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T14:02:45.466 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T14:02:45.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T14:02:45.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T14:02:45.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T14:02:45.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T14:02:45.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:02:45.467 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:02:45.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:02:45.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:02:45.467 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:02:45.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:02:45.468 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T14:02:45.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T14:02:45.469 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T14:02:45.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T14:02:45.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T14:02:45.471 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T14:02:45.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:02:45.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T14:02:45.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T14:02:45.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:02:45.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T14:02:45.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T14:02:45.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T14:02:45.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T14:02:45.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:02:45.475 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:02:45.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T14:02:45.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T14:02:45.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T14:02:45.476 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T14:02:45.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T14:02:45.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:02:45.478 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:02:45.478 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T14:02:45.480 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T14:02:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T14:02:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T14:02:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T14:02:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2024-03-09T14:02:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2024-03-09T14:02:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:983: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2024-03-09T14:02:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2024-03-09T14:02:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: list_missing_erasure_coded: local allow_overwrites=true 2024-03-09T14:02:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: list_missing_erasure_coded: local poolname=ecpool 2024-03-09T14:02:45.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:920: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2024-03-09T14:02:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T14:02:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T14:02:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T14:02:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T14:02:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T14:02:45.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T14:02:45.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T14:02:45.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:02:45.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:02:45.936 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:02:45.936 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:02:45.936 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:02:45.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:02:45.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T14:02:46.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T14:02:46.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T14:02:46.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T14:02:46.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T14:02:46.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T14:02:46.120 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T14:02:46.122 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T14:02:46.122 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T14:02:46.122 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T14:02:46.122 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T14:02:46.122 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:02:46.122 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:02:46.122 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T14:02:46.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T14:02:46.122 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T14:02:46.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T14:02:46.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T14:02:46.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T14:02:46.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T14:02:46.202 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T14:02:46.202 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T14:02:46.202 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T14:02:46.202 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T14:02:46.203 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T14:02:46.203 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:02:46.203 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:02:46.203 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T14:02:46.203 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T14:02:46.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T14:02:46.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:921: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2024-03-09T14:02:46.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T14:02:46.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T14:02:46.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T14:02:46.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T14:02:46.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T14:02:46.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T14:02:47.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T14:02:47.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:02:47.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:02:47.097 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:02:47.097 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:02:47.097 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:02:47.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:02:47.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T14:02:47.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T14:02:47.126 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: seq 0 2 2024-03-09T14:02:47.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:02:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2024-03-09T14:02:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T14:02:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T14:02:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T14:02:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T14:02:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:02:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:02:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:02:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:02:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:02:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T14:02:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T14:02:47.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T14:02:47.131 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:02:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:02:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:02:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:02:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:02:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:02:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T14:02:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T14:02:47.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T14:02:47.132 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T14:02:47.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3bbf4be9-9bc6-4aa3-804a-516d1aaa718e 2024-03-09T14:02:47.133 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3bbf4be9-9bc6-4aa3-804a-516d1aaa718e' 2024-03-09T14:02:47.133 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 3bbf4be9-9bc6-4aa3-804a-516d1aaa718e 2024-03-09T14:02:47.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T14:02:47.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHbOxlYNnZCBAAX1v4TisEbnwquToETGvPNw== 2024-03-09T14:02:47.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHbOxlYNnZCBAAX1v4TisEbnwquToETGvPNw=="}' 2024-03-09T14:02:47.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3bbf4be9-9bc6-4aa3-804a-516d1aaa718e -i td/osd-scrub-repair/0/new.json 2024-03-09T14:02:47.353 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:02:47.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T14:02:47.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHbOxlYNnZCBAAX1v4TisEbnwquToETGvPNw== --osd-uuid 3bbf4be9-9bc6-4aa3-804a-516d1aaa718e 2024-03-09T14:02:47.393 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:47.392+0000 7fc73de8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:47.404 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:47.403+0000 7fc73de8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:47.404 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:47.404+0000 7fc73de8b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:47.405 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:47.404+0000 7fc73de8b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:02:47.405 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:47.404+0000 7fc73de8b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:02:47.405 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:47.404+0000 7fc73de8b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:02:47.520 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:47.520+0000 7fc73de8b640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T14:02:53.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T14:02:53.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T14:02:53.108 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T14:02:53.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T14:02:53.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T14:02:53.468 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T14:02:53.468 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T14:02:53.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T14:02:53.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T14:02:53.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T14:02:53.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T14:02:53.494 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:53.493+0000 7f26b60c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:53.496 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:53.495+0000 7f26b60c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:53.497 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:53.496+0000 7f26b60c6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:02:53.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T14:02:53.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:02:53.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T14:02:53.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:02:53.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:02:53.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:53.742 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:02:53.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:02:53.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:53.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:02:54.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:02:54.029 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:54.028+0000 7f26b60c6640 -1 Falling back to public interface 2024-03-09T14:02:55.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:02:55.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:55.015 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:02:55.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:02:55.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:55.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:02:55.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:02:56.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:02:56.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:56.297 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:02:56.297 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:02:56.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:56.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:02:56.354 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:02:56.353+0000 7f26b60c6640 -1 osd.0 0 log_to_monitors true 2024-03-09T14:02:56.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:02:57.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:02:57.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:57.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:02:57.622 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:02:57.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:57.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:02:57.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:02:58.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:02:58.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:02:58.918 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T14:02:58.919 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T14:02:58.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:02:58.919 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:02:59.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:00.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:00.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:00.255 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T14:03:00.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T14:03:00.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:00.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:00.540 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:01.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:01.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:01.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T14:03:01.542 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T14:03:01.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:01.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:01.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:02.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:02.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:02.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T14:03:02.826 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T14:03:02.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:02.826 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:03.110 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:04.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:04.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:04.112 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T14:03:04.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T14:03:04.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:04.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:04.394 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:05.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:05.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:05.396 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T14:03:05.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T14:03:05.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:05.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:05.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:06.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:06.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:06.680 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T14:03:06.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T14:03:06.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:06.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:06.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:07.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:07.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:07.964 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T14:03:07.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T14:03:07.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:07.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:08.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:09.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:09.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:09.453 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T14:03:09.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T14:03:09.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:09.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:09.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:10.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:10.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:10.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T14:03:10.990 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T14:03:10.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:10.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:11.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:12.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:12.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:12.948 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T14:03:12.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T14:03:12.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:12.949 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:13.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:14.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:14.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:14.383 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T14:03:14.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T14:03:14.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:14.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:15.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:16.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:16.103 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:16.103 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T14:03:16.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T14:03:16.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:16.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:16.846 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:17.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:17.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:17.847 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T14:03:17.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T14:03:17.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:17.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:18.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:19.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:19.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:19.159 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T14:03:19.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T14:03:19.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:19.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:03:19.532 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2680347579,v1:127.0.0.1:6803/2680347579] [v2:127.0.0.1:6804/2680347579,v1:127.0.0.1:6805/2680347579] exists,up 3bbf4be9-9bc6-4aa3-804a-516d1aaa718e 2024-03-09T14:03:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:03:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:03:19.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T14:03:19.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T14:03:19.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:03:19.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:03:19.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:03:19.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:03:19.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:03:19.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:03:19.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T14:03:19.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T14:03:19.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T14:03:19.539 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T14:03:19.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3ada5c88-2a04-4ad3-b386-f2885d5b477c 2024-03-09T14:03:19.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3ada5c88-2a04-4ad3-b386-f2885d5b477c' 2024-03-09T14:03:19.540 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 3ada5c88-2a04-4ad3-b386-f2885d5b477c 2024-03-09T14:03:19.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T14:03:19.554 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAnbOxl0YEJIRAANtFQUN/GYKIWGVWUONZXAg== 2024-03-09T14:03:19.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAnbOxl0YEJIRAANtFQUN/GYKIWGVWUONZXAg=="}' 2024-03-09T14:03:19.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3ada5c88-2a04-4ad3-b386-f2885d5b477c -i td/osd-scrub-repair/1/new.json 2024-03-09T14:03:19.879 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:03:19.890 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T14:03:19.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAnbOxl0YEJIRAANtFQUN/GYKIWGVWUONZXAg== --osd-uuid 3ada5c88-2a04-4ad3-b386-f2885d5b477c 2024-03-09T14:03:19.920 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:19.920+0000 7ff8d3014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:03:19.955 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:19.955+0000 7ff8d3014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:03:19.956 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:19.956+0000 7ff8d3014640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:03:19.956 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:19.957+0000 7ff8d3014640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:03:19.957 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:19.957+0000 7ff8d3014640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:03:19.957 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:19.957+0000 7ff8d3014640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:03:20.023 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:20.022+0000 7ff8d3014640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T14:03:25.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T14:03:25.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T14:03:25.474 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T14:03:25.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T14:03:25.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T14:03:25.830 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T14:03:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T14:03:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T14:03:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T14:03:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T14:03:25.831 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T14:03:25.856 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:25.856+0000 7f8c5ca15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:03:25.858 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:25.858+0000 7f8c5ca15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:03:25.859 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:25.859+0000 7f8c5ca15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:03:26.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T14:03:26.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:03:26.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T14:03:26.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:03:26.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:03:26.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:26.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:03:26.108 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:03:26.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:26.108 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:26.382 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:26.382+0000 7f8c5ca15640 -1 Falling back to public interface 2024-03-09T14:03:26.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:27.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:27.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:27.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:03:27.423 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:03:27.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:27.423 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:27.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:28.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:28.672 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:03:28.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:28.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:03:28.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:28.672 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:28.729 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:28.729+0000 7f8c5ca15640 -1 osd.1 0 log_to_monitors true 2024-03-09T14:03:28.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:29.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:29.958 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:03:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:03:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:29.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:30.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:31.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:31.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:31.242 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T14:03:31.243 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T14:03:31.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:31.243 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:31.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:32.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:32.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:32.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T14:03:32.575 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T14:03:32.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:32.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:32.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:33.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:33.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:33.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T14:03:33.859 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T14:03:33.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:33.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:34.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:35.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:35.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:35.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T14:03:35.143 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T14:03:35.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:35.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:35.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:36.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:36.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:36.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T14:03:36.525 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T14:03:36.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:36.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:36.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:37.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:37.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:37.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T14:03:37.811 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T14:03:37.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:37.811 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:38.095 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:39.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:39.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:39.098 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T14:03:39.098 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T14:03:39.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:39.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:39.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:40.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:40.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:40.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T14:03:40.566 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T14:03:40.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:40.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:41.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:42.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:42.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:42.022 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T14:03:42.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T14:03:42.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:42.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:42.689 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:43.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:43.690 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:43.691 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T14:03:43.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T14:03:43.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:43.691 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:44.115 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:45.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:45.116 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:45.116 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T14:03:45.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T14:03:45.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:45.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:45.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:46.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:46.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:46.593 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T14:03:46.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T14:03:46.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:46.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:47.238 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:48.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:48.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:48.240 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T14:03:48.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T14:03:48.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:48.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:48.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:49.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:49.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:49.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T14:03:49.593 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T14:03:49.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:49.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:49.908 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:50.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:50.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:50.910 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T14:03:50.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T14:03:50.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:50.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:51.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:03:52.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:03:52.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:52.452 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T14:03:52.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T14:03:52.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:52.453 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:03:52.731 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2575254831,v1:127.0.0.1:6811/2575254831] [v2:127.0.0.1:6812/2575254831,v1:127.0.0.1:6813/2575254831] exists,up 3ada5c88-2a04-4ad3-b386-f2885d5b477c 2024-03-09T14:03:52.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:03:52.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:03:52.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:03:52.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:03:52.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:923: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2024-03-09T14:03:52.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:03:52.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T14:03:52.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:03:52.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:03:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:03:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:03:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:03:52.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:03:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:03:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:03:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T14:03:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T14:03:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:03:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:03:52.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:03:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:03:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:03:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:03:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T14:03:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-03-09T14:03:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T14:03:52.739 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T14:03:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=da01aba0-2850-4129-8e7a-6d8e17bc912d 2024-03-09T14:03:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 da01aba0-2850-4129-8e7a-6d8e17bc912d' 2024-03-09T14:03:52.739 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 da01aba0-2850-4129-8e7a-6d8e17bc912d 2024-03-09T14:03:52.740 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T14:03:52.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIbOxlSETpLBAA72QgagkS3lk/djjYMwmiXQ== 2024-03-09T14:03:52.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIbOxlSETpLBAA72QgagkS3lk/djjYMwmiXQ=="}' 2024-03-09T14:03:52.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new da01aba0-2850-4129-8e7a-6d8e17bc912d -i td/osd-scrub-repair/2/new.json 2024-03-09T14:03:53.162 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:03:53.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T14:03:53.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIbOxlSETpLBAA72QgagkS3lk/djjYMwmiXQ== --osd-uuid da01aba0-2850-4129-8e7a-6d8e17bc912d 2024-03-09T14:03:53.202 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:53.201+0000 7f9b8f96f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:03:53.238 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:53.238+0000 7f9b8f96f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:03:53.239 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:53.239+0000 7f9b8f96f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:03:53.240 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:53.239+0000 7f9b8f96f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T14:03:53.240 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:53.239+0000 7f9b8f96f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T14:03:53.240 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:53.239+0000 7f9b8f96f640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T14:03:53.321 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:53.320+0000 7f9b8f96f640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T14:03:59.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T14:03:59.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T14:03:59.292 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T14:03:59.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T14:03:59.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T14:03:59.647 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T14:03:59.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T14:03:59.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T14:03:59.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T14:03:59.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T14:03:59.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T14:03:59.674 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:59.673+0000 7fdd9b264640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:03:59.675 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:59.675+0000 7fdd9b264640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:03:59.676 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:03:59.676+0000 7fdd9b264640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:03:59.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T14:03:59.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:03:59.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T14:03:59.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:03:59.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:03:59.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:03:59.921 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:03:59.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:03:59.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:03:59.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:00.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:00.207 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:04:00.207+0000 7fdd9b264640 -1 Falling back to public interface 2024-03-09T14:04:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:04:01.193 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:04:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:01.193 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:01.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:02.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:02.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:02.499 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:04:02.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:04:02.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:02.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:02.555 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:04:02.555+0000 7fdd9b264640 -1 osd.2 0 log_to_monitors true 2024-03-09T14:04:02.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:03.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:03.787 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:03.787 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:04:03.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:04:03.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:03.788 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:04.076 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:05.074 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:05.074 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T14:04:05.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:05.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T14:04:05.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:05.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:05.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:06.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:06.435 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:06.435 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T14:04:06.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T14:04:06.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:06.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:06.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:07.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:07.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:07.716 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T14:04:07.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T14:04:07.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:07.717 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:08.017 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:09.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:09.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:09.018 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T14:04:09.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T14:04:09.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:09.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:09.434 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:10.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:10.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:10.436 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T14:04:10.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T14:04:10.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:10.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:10.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:11.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:11.719 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:11.719 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T14:04:11.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T14:04:11.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:11.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:12.028 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:13.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:13.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:13.030 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T14:04:13.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T14:04:13.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:13.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:13.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:14.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:14.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:14.499 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T14:04:14.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T14:04:14.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:14.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:14.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:15.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:15.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:15.781 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T14:04:15.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T14:04:15.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:15.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:16.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:17.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:17.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:17.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T14:04:17.256 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T14:04:17.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:17.257 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:18.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:19.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:19.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:19.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T14:04:19.008 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T14:04:19.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:19.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:19.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:20.752 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:20.752 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T14:04:20.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:20.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T14:04:20.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:20.753 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:21.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:22.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:22.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:22.268 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T14:04:22.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T14:04:22.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:22.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:22.764 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:23.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:23.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:23.766 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T14:04:23.766 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T14:04:23.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:23.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:24.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:25.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:25.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:25.302 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T14:04:25.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T14:04:25.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:25.303 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:25.587 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:26.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:26.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:26.589 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T14:04:26.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T14:04:26.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:26.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:04:26.871 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/381217787,v1:127.0.0.1:6819/381217787] [v2:127.0.0.1:6820/381217787,v1:127.0.0.1:6821/381217787] exists,up da01aba0-2850-4129-8e7a-6d8e17bc912d 2024-03-09T14:04:26.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:04:26.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:04:26.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:04:26.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:925: list_missing_erasure_coded: create_rbd_pool 2024-03-09T14:04:26.872 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T14:04:27.196 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T14:04:27.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T14:04:27.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T14:04:27.664 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T14:04:27.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T14:04:28.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T14:04:31.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: list_missing_erasure_coded: wait_for_clean 2024-03-09T14:04:31.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:04:31.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:04:31.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:04:31.014 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:04:31.015 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:04:31.016 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:04:31.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:04:31.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:04:31.016 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:04:31.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:04:31.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:04:31.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:04:31.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:04:31.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:04:31.117 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:04:31.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:04:31.392 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:04:31.392 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T14:04:31.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:04:31.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:04:31.393 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:04:31.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-03-09T14:04:31.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-03-09T14:04:31.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-03-09T14:04:31.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:04:31.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:04:31.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-03-09T14:04:31.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-03-09T14:04:31.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672974' 2024-03-09T14:04:31.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:04:31.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T14:04:31.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509444 2024-03-09T14:04:31.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509444 2024-03-09T14:04:31.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672974 2-64424509444' 2024-03-09T14:04:31.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:04:31.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-03-09T14:04:31.705 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:04:31.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:04:31.706 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-03-09T14:04:31.707 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:04:31.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-03-09T14:04:31.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-03-09T14:04:31.708 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836500 2024-03-09T14:04:31.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:04:31.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836499 -lt 21474836500 2024-03-09T14:04:31.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:04:32.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:04:32.984 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:04:33.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-03-09T14:04:33.271 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:04:33.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-03-09T14:04:33.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:04:33.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:04:33.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-03-09T14:04:33.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:04:33.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-03-09T14:04:33.274 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-03-09T14:04:33.275 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672974 2024-03-09T14:04:33.275 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:04:33.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672974 2024-03-09T14:04:33.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:04:33.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509444 2024-03-09T14:04:33.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:04:33.549 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T14:04:33.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509444 2024-03-09T14:04:33.550 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:04:33.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509444 2024-03-09T14:04:33.552 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509444 2024-03-09T14:04:33.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-03-09T14:04:33.552 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T14:04:33.868 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509444 -lt 64424509444 2024-03-09T14:04:33.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:04:33.868 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:04:33.869 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:04:34.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T14:04:34.212 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:04:34.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:04:34.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:04:34.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:04:34.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:04:34.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:04:34.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:04:34.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T14:04:34.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:04:34.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:04:34.483 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:04:34.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T14:04:34.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:04:34.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:04:34.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2024-03-09T14:04:34.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=ecpool 2024-03-09T14:04:34.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T14:04:34.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=true 2024-03-09T14:04:34.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T14:04:34.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-09T14:04:35.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2024-03-09T14:04:35.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2024-03-09T14:04:35.869 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'ecpool' created 2024-03-09T14:04:35.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T14:04:36.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' true = true ']' 2024-03-09T14:04:36.883 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2024-03-09T14:04:37.535 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 2 allow_ec_overwrites to true 2024-03-09T14:04:37.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T14:04:37.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:04:37.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:04:37.548 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:04:37.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:04:37.549 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:04:37.549 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:04:37.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:04:37.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:04:37.549 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:04:37.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:04:37.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:04:37.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:04:37.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:04:37.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:04:37.641 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:04:37.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:04:37.992 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:04:37.992 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T14:04:37.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:04:37.992 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:04:37.992 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:04:38.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-03-09T14:04:38.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-03-09T14:04:38.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-03-09T14:04:38.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:04:38.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:04:38.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-03-09T14:04:38.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-03-09T14:04:38.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672977' 2024-03-09T14:04:38.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:04:38.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T14:04:38.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-03-09T14:04:38.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-03-09T14:04:38.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672977 2-64424509446' 2024-03-09T14:04:38.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:04:38.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-03-09T14:04:38.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:04:38.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:04:38.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-03-09T14:04:38.318 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:04:38.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-03-09T14:04:38.319 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-03-09T14:04:38.319 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836502 2024-03-09T14:04:38.320 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:04:38.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-03-09T14:04:38.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:04:39.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:04:39.594 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:04:39.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836503 -lt 21474836502 2024-03-09T14:04:39.877 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:04:39.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-03-09T14:04:39.878 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:04:39.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:04:39.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-03-09T14:04:39.880 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:04:39.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-03-09T14:04:39.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-03-09T14:04:39.881 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672977 2024-03-09T14:04:39.882 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:04:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-03-09T14:04:40.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:04:40.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-03-09T14:04:40.153 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:04:40.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T14:04:40.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-03-09T14:04:40.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:04:40.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-03-09T14:04:40.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-03-09T14:04:40.157 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509446 2024-03-09T14:04:40.157 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T14:04:40.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-03-09T14:04:40.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:04:40.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:04:40.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:04:40.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T14:04:40.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:04:40.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:04:40.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:04:40.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:04:40.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:04:40.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:04:40.769 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:04:41.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T14:04:41.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:04:41.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:04:41.039 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:04:41.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T14:04:41.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:04:41.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:04:41.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T14:04:41.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:931: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2024-03-09T14:04:41.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T14:04:41.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T14:04:41.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2024-03-09T14:04:41.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T14:04:41.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T14:04:41.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T14:04:41.744 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T14:04:41.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T14:04:42.257 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T14:04:42.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T14:04:42.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T14:04:42.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:04:42.579 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: get_osds ecpool MOBJ0 2024-03-09T14:04:42.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T14:04:42.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ0 2024-03-09T14:04:42.580 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ0 2024-03-09T14:04:42.580 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T14:04:42.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-09T14:04:42.885 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:04:42.885 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T14:04:42.885 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-09T14:04:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: osds0=('2' '1' '0') 2024-03-09T14:04:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:932: list_missing_erasure_coded: local -a osds0 2024-03-09T14:04:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2024-03-09T14:04:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T14:04:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2024-03-09T14:04:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2024-03-09T14:04:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2024-03-09T14:04:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2024-03-09T14:04:42.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2024-03-09T14:04:43.347 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is set 2024-03-09T14:04:43.358 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2024-03-09T14:04:43.858 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T14:04:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T14:04:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T14:04:43.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:04:44.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: get_osds ecpool MOBJ1 2024-03-09T14:04:44.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-03-09T14:04:44.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=MOBJ1 2024-03-09T14:04:44.113 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool MOBJ1 2024-03-09T14:04:44.113 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-03-09T14:04:44.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-03-09T14:04:44.417 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:04:44.417 INFO:tasks.workunit.client.0.smithi162.stderr:0' 2024-03-09T14:04:44.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-03-09T14:04:44.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: osds1=('2' '1' '0') 2024-03-09T14:04:44.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: list_missing_erasure_coded: local -a osds1 2024-03-09T14:04:44.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: seq 0 2 2024-03-09T14:04:44.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:04:44.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T14:04:44.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:04:44.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:04:44.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:04:44.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:04:44.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:04:44.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:04:44.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:04:44.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T14:04:44.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:04:44.727 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:04:44.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:04:44.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:04:44.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:04:46.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:04:46.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:04:46.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2024-03-09T14:04:46.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:04:46.036 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:04:46.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:04:46.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:04:46.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:04:46.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:04:46.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:943: list_missing_erasure_coded: id=2 2024-03-09T14:04:46.346 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:944: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2024-03-09T14:04:47.542 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#2:a9030505:::MOBJ0:head# 2024-03-09T14:04:48.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: list_missing_erasure_coded: id=1 2024-03-09T14:04:48.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2024-03-09T14:04:49.121 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#2:a9030505:::MOBJ0:head# 2024-03-09T14:04:49.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:950: list_missing_erasure_coded: id=1 2024-03-09T14:04:49.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:951: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2024-03-09T14:04:50.383 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2024-03-09T14:04:50.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: list_missing_erasure_coded: id=0 2024-03-09T14:04:50.898 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:954: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2024-03-09T14:04:52.062 INFO:tasks.workunit.client.0.smithi162.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2024-03-09T14:04:52.576 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: seq 0 2 2024-03-09T14:04:52.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:04:52.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2024-03-09T14:04:52.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T14:04:52.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T14:04:52.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T14:04:52.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T14:04:52.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:04:52.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:04:52.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:04:52.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:04:52.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:04:52.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T14:04:52.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T14:04:52.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T14:04:52.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T14:04:52.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:04:52.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T14:04:52.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:04:52.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:04:52.582 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:04:52.582 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:04:52.582 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:04:52.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:04:52.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:04:52.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:04:52.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:04:52.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:04:52.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:04:52.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:04:52.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:04:52.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:04:52.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T14:04:52.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T14:04:52.583 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T14:04:52.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T14:04:52.584 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T14:04:52.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T14:04:52.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T14:04:52.584 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T14:04:52.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T14:04:52.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T14:04:52.585 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T14:04:52.614 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:04:52.611+0000 7f65ce5bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:04:52.614 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:04:52.613+0000 7f65ce5bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:04:52.614 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:04:52.614+0000 7f65ce5bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:04:52.866 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T14:04:52.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:04:52.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T14:04:52.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:04:52.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:04:52.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:52.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:04:52.867 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T14:04:52.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:52.867 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:04:53.131 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:04:53.132+0000 7f65ce5bb640 -1 Falling back to public interface 2024-03-09T14:04:53.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:54.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:54.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:54.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:04:54.143 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:04:54.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:54.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:04:54.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:55.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:55.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:55.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:04:55.424 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T14:04:55.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:55.424 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:04:55.659 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:04:55.659+0000 7f65ce5bb640 -1 osd.0 35 log_to_monitors true 2024-03-09T14:04:55.699 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:56.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:56.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:56.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:04:56.701 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T14:04:56.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:56.701 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:04:56.981 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 41 up_thru 41 down_at 36 last_clean_interval [5,35) [v2:127.0.0.1:6802/1671663368,v1:127.0.0.1:6803/1671663368] [v2:127.0.0.1:6804/1671663368,v1:127.0.0.1:6805/1671663368] exists,up 3bbf4be9-9bc6-4aa3-804a-516d1aaa718e 2024-03-09T14:04:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:04:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:04:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:04:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:04:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2024-03-09T14:04:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T14:04:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T14:04:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T14:04:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T14:04:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:04:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:04:56.982 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:04:56.983 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:04:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:04:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:04:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:04:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:04:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:04:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:04:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:04:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T14:04:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T14:04:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T14:04:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T14:04:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.1 2024-03-09T14:04:56.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T14:04:56.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T14:04:56.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T14:04:56.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T14:04:56.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T14:04:56.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T14:04:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:04:57.007+0000 7fe17cc64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:04:57.008 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:04:57.009+0000 7fe17cc64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:04:57.009 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:04:57.010+0000 7fe17cc64640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:04:57.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T14:04:57.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:04:57.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T14:04:57.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:04:57.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:04:57.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:57.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:04:57.260 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T14:04:57.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:57.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:04:57.531 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:04:57.532+0000 7fe17cc64640 -1 Falling back to public interface 2024-03-09T14:04:57.535 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:58.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:58.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:58.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:04:58.537 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:04:58.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:58.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:04:58.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:04:59.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:04:59.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:04:59.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:04:59.823 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T14:04:59.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:04:59.823 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:04:59.837 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:04:59.837+0000 7fe17cc64640 -1 osd.1 36 log_to_monitors true 2024-03-09T14:05:00.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:01.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:01.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:01.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:05:01.134 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T14:05:01.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:01.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:05:01.436 INFO:tasks.workunit.client.0.smithi162.stderr:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 37 last_clean_interval [10,36) [v2:127.0.0.1:6810/647528179,v1:127.0.0.1:6811/647528179] [v2:127.0.0.1:6812/647528179,v1:127.0.0.1:6813/647528179] exists,up 3ada5c88-2a04-4ad3-b386-f2885d5b477c 2024-03-09T14:05:01.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:05:01.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:05:01.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:05:01.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: list_missing_erasure_coded: for id in $(seq 0 2) 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T14:05:01.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:05:01.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:05:01.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:05:01.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T14:05:01.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-03-09T14:05:01.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T14:05:01.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-03-09T14:05:01.439 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.2 2024-03-09T14:05:01.439 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-03-09T14:05:01.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/2/whoami 2024-03-09T14:05:01.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-03-09T14:05:01.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T14:05:01.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T14:05:01.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T14:05:01.461 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:01.461+0000 7f0d20a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:01.463 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:01.463+0000 7f0d20a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:01.464 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:01.464+0000 7f0d20a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:01.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-03-09T14:05:01.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:05:01.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T14:05:01.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:05:01.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:05:01.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:01.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:05:01.711 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T14:05:01.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:01.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:05:01.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:01.992 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:01.993+0000 7f0d20a15640 -1 Falling back to public interface 2024-03-09T14:05:02.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:02.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:02.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:05:02.989 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:05:02.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:02.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:05:03.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:05:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T14:05:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:04.302 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:05:04.350 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:04.350+0000 7f0d20a15640 -1 osd.2 38 log_to_monitors true 2024-03-09T14:05:04.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:05.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:05.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:05.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:05:05.591 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T14:05:05.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:05.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:05:05.902 INFO:tasks.workunit.client.0.smithi162.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 39 last_clean_interval [15,38) [v2:127.0.0.1:6818/906383673,v1:127.0.0.1:6819/906383673] [v2:127.0.0.1:6820/906383673,v1:127.0.0.1:6821/906383673] exists,up da01aba0-2850-4129-8e7a-6d8e17bc912d 2024-03-09T14:05:05.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:05:05.902 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:05:05.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:05:05.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:960: list_missing_erasure_coded: create_rbd_pool 2024-03-09T14:05:05.903 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T14:05:06.452 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' removed 2024-03-09T14:05:06.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T14:05:06.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T14:05:06.837 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T14:05:06.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T14:05:07.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T14:05:09.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:961: list_missing_erasure_coded: wait_for_clean 2024-03-09T14:05:09.153 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:05:09.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:05:09.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:05:09.154 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:05:09.154 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:05:09.155 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:05:09.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:05:09.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:05:09.155 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:05:09.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:05:09.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:05:09.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:05:09.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:05:09.256 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:05:09.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:05:09.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:05:09.530 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:05:09.530 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T14:05:09.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:05:09.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:05:09.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:05:09.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659140 2024-03-09T14:05:09.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659140 2024-03-09T14:05:09.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140' 2024-03-09T14:05:09.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:05:09.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:05:09.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=188978561027 2024-03-09T14:05:09.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 188978561027 2024-03-09T14:05:09.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140 1-188978561027' 2024-03-09T14:05:09.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:05:09.738 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T14:05:09.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462915 2024-03-09T14:05:09.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462915 2024-03-09T14:05:09.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-176093659140 1-188978561027 2-201863462915' 2024-03-09T14:05:09.841 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:05:09.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-176093659140 2024-03-09T14:05:09.841 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:05:09.842 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:05:09.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-176093659140 2024-03-09T14:05:09.843 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:05:09.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659140 2024-03-09T14:05:09.844 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 176093659140' 2024-03-09T14:05:09.844 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 176093659140 2024-03-09T14:05:09.844 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:05:10.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659139 -lt 176093659140 2024-03-09T14:05:10.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:05:11.118 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:05:11.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:05:11.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659141 -lt 176093659140 2024-03-09T14:05:11.417 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:05:11.417 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-188978561027 2024-03-09T14:05:11.418 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:05:11.418 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:05:11.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-188978561027 2024-03-09T14:05:11.419 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:05:11.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=188978561027 2024-03-09T14:05:11.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 188978561027' 2024-03-09T14:05:11.421 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 188978561027 2024-03-09T14:05:11.421 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:05:11.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 188978561028 -lt 188978561027 2024-03-09T14:05:11.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:05:11.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-201863462915 2024-03-09T14:05:11.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:05:11.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T14:05:11.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-201863462915 2024-03-09T14:05:11.697 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:05:11.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462915 2024-03-09T14:05:11.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 201863462915' 2024-03-09T14:05:11.698 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 201863462915 2024-03-09T14:05:11.698 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T14:05:11.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462915 -lt 201863462915 2024-03-09T14:05:11.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:05:11.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:05:11.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:05:12.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T14:05:12.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:05:12.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:05:12.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:05:12.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:05:12.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:05:12.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:05:12.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:05:12.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T14:05:12.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:05:12.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:05:12.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:05:12.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T14:05:12.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:05:12.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:05:12.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: get_pg ecpool MOBJ0 2024-03-09T14:05:12.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=ecpool 2024-03-09T14:05:12.917 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=MOBJ0 2024-03-09T14:05:12.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map ecpool MOBJ0 2024-03-09T14:05:12.918 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T14:05:13.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:964: list_missing_erasure_coded: local pg=2.0 2024-03-09T14:05:13.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:968: list_missing_erasure_coded: repair 2.0 2024-03-09T14:05:13.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=2.0 2024-03-09T14:05:13.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 2.0 2024-03-09T14:05:13.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:05:13.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:05:13.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:05:13.241 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:05:13.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T14:04:35.735845+0000 2024-03-09T14:05:13.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 2.0 2024-03-09T14:05:13.786 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0s0 on osd.2 to repair 2024-03-09T14:05:13.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 2.0 2024-03-09T14:04:35.735845+0000 2024-03-09T14:05:13.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T14:05:13.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T14:04:35.735845+0000 2024-03-09T14:05:13.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T14:05:13.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T14:05:13.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:05:13.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T14:05:13.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:05:13.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:05:13.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:05:13.798 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:05:14.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:04:35.735845+0000 '>' 2024-03-09T14:04:35.735845+0000 2024-03-09T14:05:14.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:05:15.071 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:05:15.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:05:15.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T14:05:15.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:05:15.072 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:05:15.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:05:15.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:05:15.375 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:05:14.369714+0000 '>' 2024-03-09T14:04:35.735845+0000 2024-03-09T14:05:15.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T14:05:15.376 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: seq 0 120 2024-03-09T14:05:15.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:970: list_missing_erasure_coded: for i in $(seq 0 120) 2024-03-09T14:05:15.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:971: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2024-03-09T14:05:15.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2024-03-09T14:05:15.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2024-03-09T14:05:15.378 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: wc -l 2024-03-09T14:05:15.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:972: list_missing_erasure_coded: matches=2 2024-03-09T14:05:15.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2024-03-09T14:05:15.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:973: list_missing_erasure_coded: break 2024-03-09T14:05:15.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T14:05:15.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T14:05:15.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T14:05:15.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T14:05:15.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:05:15.487 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:05:15.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:05:15.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:05:15.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:05:15.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:05:15.596 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T14:05:15.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T14:05:15.597 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T14:05:15.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T14:05:15.598 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T14:05:15.598 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T14:05:15.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:05:15.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T14:05:15.599 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T14:05:15.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:05:15.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T14:05:15.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T14:05:15.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T14:05:15.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T14:05:15.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:05:15.618 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:05:15.618 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T14:05:15.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T14:05:15.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T14:05:15.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T14:05:15.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T14:05:15.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T14:05:15.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T14:05:15.620 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T14:05:15.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T14:05:15.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T14:05:15.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:05:15.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:05:15.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:05:15.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:05:15.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:05:15.622 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:05:15.622 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T14:05:15.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T14:05:15.624 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T14:05:15.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T14:05:15.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T14:05:15.625 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T14:05:15.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:05:15.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T14:05:15.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T14:05:15.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:05:15.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T14:05:15.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T14:05:15.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T14:05:15.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T14:05:15.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:05:15.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:05:15.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T14:05:15.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T14:05:15.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T14:05:15.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T14:05:15.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T14:05:15.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:05:15.632 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:05:15.632 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T14:05:15.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T14:05:15.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T14:05:15.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T14:05:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T14:05:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2024-03-09T14:05:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5759: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2024-03-09T14:05:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5760: TEST_periodic_scrub_replicated: local poolname=psr_pool 2024-03-09T14:05:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5761: TEST_periodic_scrub_replicated: local objname=POBJ 2024-03-09T14:05:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5763: TEST_periodic_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2024-03-09T14:05:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T14:05:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T14:05:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T14:05:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T14:05:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T14:05:15.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2024-03-09T14:05:16.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T14:05:16.096 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:05:16.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:05:16.097 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:05:16.097 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:05:16.097 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:05:16.097 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:05:16.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-03-09T14:05:16.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T14:05:16.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T14:05:16.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T14:05:16.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T14:05:16.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T14:05:16.280 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T14:05:16.281 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T14:05:16.281 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T14:05:16.281 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T14:05:16.281 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T14:05:16.281 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:05:16.282 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:05:16.282 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T14:05:16.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T14:05:16.282 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T14:05:16.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T14:05:16.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T14:05:16.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T14:05:16.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T14:05:16.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T14:05:16.365 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T14:05:16.365 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T14:05:16.365 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T14:05:16.366 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T14:05:16.366 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:05:16.366 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:05:16.366 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T14:05:16.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T14:05:16.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T14:05:16.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5764: TEST_periodic_scrub_replicated: run_mgr td/osd-scrub-repair x 2024-03-09T14:05:16.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T14:05:16.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T14:05:16.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T14:05:16.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T14:05:16.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T14:05:16.444 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T14:05:17.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T14:05:17.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:05:17.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:05:17.257 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:05:17.257 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:05:17.257 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:05:17.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:05:17.257 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T14:05:17.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T14:05:17.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5765: TEST_periodic_scrub_replicated: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2024-03-09T14:05:17.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5766: TEST_periodic_scrub_replicated: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2024-03-09T14:05:17.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5767: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-09T14:05:17.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T14:05:17.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T14:05:17.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T14:05:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T14:05:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:05:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:05:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:05:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:05:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:05:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T14:05:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T14:05:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T14:05:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T14:05:17.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:05:17.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T14:05:17.290 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:05:17.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:05:17.291 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:05:17.291 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:05:17.291 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:05:17.291 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:05:17.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:05:17.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:05:17.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T14:05:17.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T14:05:17.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:05:17.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:05:17.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:05:17.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:05:17.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:05:17.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:05:17.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T14:05:17.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-09T14:05:17.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T14:05:17.292 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T14:05:17.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=47219909-4497-48c1-aef2-920fb8a580ae 2024-03-09T14:05:17.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 47219909-4497-48c1-aef2-920fb8a580ae' 2024-03-09T14:05:17.293 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 47219909-4497-48c1-aef2-920fb8a580ae 2024-03-09T14:05:17.293 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T14:05:17.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdbOxlrBBDEhAA9KB6m+ZUb3vpLOmkRfTVbQ== 2024-03-09T14:05:17.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdbOxlrBBDEhAA9KB6m+ZUb3vpLOmkRfTVbQ=="}' 2024-03-09T14:05:17.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 47219909-4497-48c1-aef2-920fb8a580ae -i td/osd-scrub-repair/0/new.json 2024-03-09T14:05:17.512 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:05:17.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T14:05:17.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQCdbOxlrBBDEhAA9KB6m+ZUb3vpLOmkRfTVbQ== --osd-uuid 47219909-4497-48c1-aef2-920fb8a580ae 2024-03-09T14:05:17.554 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:17.553+0000 7f2497f71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:17.563 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:17.562+0000 7f2497f71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:17.564 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:17.563+0000 7f2497f71640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:17.564 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:17.564+0000 7f2497f71640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:05:17.564 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:17.564+0000 7f2497f71640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:05:17.565 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:17.564+0000 7f2497f71640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:05:17.655 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:17.654+0000 7f2497f71640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T14:05:23.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T14:05:23.239 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T14:05:23.240 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T14:05:23.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T14:05:23.240 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T14:05:23.593 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T14:05:23.594 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T14:05:23.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-09T14:05:23.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T14:05:23.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T14:05:23.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T14:05:23.622 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:23.621+0000 7f642cf21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:23.628 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:23.627+0000 7f642cf21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:23.629 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:23.628+0000 7f642cf21640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:23.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T14:05:23.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:05:23.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T14:05:23.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:05:23.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:05:23.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:23.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:05:23.892 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:05:23.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:23.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:24.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:24.168 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:24.167+0000 7f642cf21640 -1 Falling back to public interface 2024-03-09T14:05:25.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:25.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:25.164 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:05:25.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:05:25.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:25.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:25.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:26.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:26.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:26.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:05:26.440 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:05:26.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:26.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:26.528 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:26.526+0000 7f642cf21640 -1 osd.0 0 log_to_monitors true 2024-03-09T14:05:26.718 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:27.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:27.720 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:05:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:05:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:27.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:28.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:29.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:29.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:29.004 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T14:05:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T14:05:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:29.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:29.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:30.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:30.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:30.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T14:05:30.288 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T14:05:30.288 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:30.289 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:30.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:31.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:31.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:31.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T14:05:31.575 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T14:05:31.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:31.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:31.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:32.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:32.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:32.859 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T14:05:32.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T14:05:32.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:32.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:33.141 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:34.142 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:34.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:34.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T14:05:34.143 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T14:05:34.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:34.144 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:34.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:35.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:35.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:35.428 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T14:05:35.428 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T14:05:35.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:35.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:35.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:36.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:36.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:36.710 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T14:05:36.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T14:05:36.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:36.710 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:36.994 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:37.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:37.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:37.995 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T14:05:37.995 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T14:05:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:37.996 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:38.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:39.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:39.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:39.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T14:05:39.332 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T14:05:39.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:39.333 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:39.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:40.648 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:40.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:40.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T14:05:40.649 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T14:05:40.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:40.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:41.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:42.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:42.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:42.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T14:05:42.185 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T14:05:42.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:42.186 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:42.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:43.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:43.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:43.679 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T14:05:43.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T14:05:43.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:43.679 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:44.721 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:45.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:45.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:45.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T14:05:45.723 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T14:05:45.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:45.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:46.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:47.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:47.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:47.183 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T14:05:47.183 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T14:05:47.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:47.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:47.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:48.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:48.577 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T14:05:48.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:48.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T14:05:48.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:48.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:48.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:49.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:49.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:49.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T14:05:49.862 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T14:05:49.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:49.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:05:50.142 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2934236132,v1:127.0.0.1:6803/2934236132] [v2:127.0.0.1:6804/2934236132,v1:127.0.0.1:6805/2934236132] exists,up 47219909-4497-48c1-aef2-920fb8a580ae 2024-03-09T14:05:50.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:05:50.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:05:50.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:05:50.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5768: TEST_periodic_scrub_replicated: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:05:50.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:05:50.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:05:50.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:05:50.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:05:50.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T14:05:50.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-09T14:05:50.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T14:05:50.149 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T14:05:50.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f16fb69b-e1fa-4e92-a5b7-5296094df042 2024-03-09T14:05:50.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f16fb69b-e1fa-4e92-a5b7-5296094df042' 2024-03-09T14:05:50.149 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 f16fb69b-e1fa-4e92-a5b7-5296094df042 2024-03-09T14:05:50.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T14:05:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC+bOxlvjHKCRAAiFoBG5JkTB3NPL84WvgVLQ== 2024-03-09T14:05:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC+bOxlvjHKCRAAiFoBG5JkTB3NPL84WvgVLQ=="}' 2024-03-09T14:05:50.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f16fb69b-e1fa-4e92-a5b7-5296094df042 -i td/osd-scrub-repair/1/new.json 2024-03-09T14:05:50.498 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:05:50.509 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T14:05:50.510 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQC+bOxlvjHKCRAAiFoBG5JkTB3NPL84WvgVLQ== --osd-uuid f16fb69b-e1fa-4e92-a5b7-5296094df042 2024-03-09T14:05:50.541 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:50.541+0000 7f5536889640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:50.582 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:50.582+0000 7f5536889640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:50.583 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:50.583+0000 7f5536889640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:50.583 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:50.584+0000 7f5536889640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:05:50.583 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:50.584+0000 7f5536889640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:05:50.584 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:50.584+0000 7f5536889640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:05:50.649 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:50.649+0000 7f5536889640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T14:05:56.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T14:05:56.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T14:05:56.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T14:05:56.706 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T14:05:56.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T14:05:57.064 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T14:05:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T14:05:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-09T14:05:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T14:05:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T14:05:57.065 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T14:05:57.092 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:57.093+0000 7f24b56bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:57.094 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:57.095+0000 7f24b56bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:57.095 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:57.096+0000 7f24b56bd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:05:57.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T14:05:57.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:05:57.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T14:05:57.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:05:57.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:05:57.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:57.342 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:05:57.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:05:57.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:57.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:05:57.614 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:57.614+0000 7f24b56bd640 -1 Falling back to public interface 2024-03-09T14:05:57.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:58.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:58.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:58.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:05:58.630 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:05:58.630 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:58.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:05:58.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:05:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:05:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:05:59.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:05:59.910 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:05:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:05:59.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:05:59.962 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:05:59.961+0000 7f24b56bd640 -1 osd.1 0 log_to_monitors true 2024-03-09T14:06:00.190 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:01.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:01.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:01.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:06:01.191 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:06:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:01.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:01.472 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:02.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:02.474 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:02.474 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T14:06:02.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T14:06:02.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:02.475 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:02.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:03.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:03.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:03.780 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T14:06:03.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T14:06:03.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:03.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:04.064 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:05.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:05.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:05.066 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T14:06:05.066 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T14:06:05.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:05.067 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:05.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:06.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:06.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:06.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T14:06:06.363 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T14:06:06.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:06.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:06.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:07.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:07.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:07.744 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T14:06:07.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T14:06:07.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:07.744 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:08.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:09.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:09.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:09.049 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T14:06:09.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T14:06:09.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:09.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:09.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:10.352 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:10.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:10.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T14:06:10.353 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T14:06:10.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:10.353 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:10.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:11.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:11.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:11.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T14:06:11.760 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T14:06:11.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:11.760 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:12.244 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:13.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:13.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:13.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T14:06:13.247 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T14:06:13.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:13.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:13.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:14.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:14.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:14.989 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T14:06:14.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T14:06:14.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:14.990 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:15.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:16.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:16.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:16.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T14:06:16.522 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T14:06:16.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:16.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:16.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:17.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T14:06:17.850 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T14:06:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:17.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:18.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:19.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:19.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:19.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T14:06:19.594 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T14:06:19.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:19.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:20.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:21.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:21.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:21.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T14:06:21.216 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T14:06:21.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:21.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:21.741 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:22.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:22.742 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:22.742 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T14:06:22.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T14:06:22.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:22.743 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:06:23.020 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1430040088,v1:127.0.0.1:6811/1430040088] [v2:127.0.0.1:6812/1430040088,v1:127.0.0.1:6813/1430040088] exists,up f16fb69b-e1fa-4e92-a5b7-5296094df042 2024-03-09T14:06:23.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:06:23.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:06:23.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:06:23.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5769: TEST_periodic_scrub_replicated: create_rbd_pool 2024-03-09T14:06:23.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-03-09T14:06:23.301 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' does not exist 2024-03-09T14:06:23.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-03-09T14:06:23.313 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-03-09T14:06:23.725 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'rbd' created 2024-03-09T14:06:23.736 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T14:06:24.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-03-09T14:06:26.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5770: TEST_periodic_scrub_replicated: wait_for_clean 2024-03-09T14:06:26.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:06:26.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:06:26.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:06:26.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:06:26.528 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:06:26.528 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:06:26.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:06:26.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:06:26.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:06:26.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:06:26.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:06:26.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:06:26.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:06:26.629 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:06:26.629 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:06:26.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:06:26.901 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T14:06:26.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:06:26.901 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:06:26.901 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:06:27.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836494 2024-03-09T14:06:27.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836494 2024-03-09T14:06:27.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494' 2024-03-09T14:06:27.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:06:27.005 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:06:27.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672966 2024-03-09T14:06:27.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672966 2024-03-09T14:06:27.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836494 1-42949672966' 2024-03-09T14:06:27.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:06:27.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836494 2024-03-09T14:06:27.110 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:06:27.111 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:06:27.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836494 2024-03-09T14:06:27.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:06:27.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836494 2024-03-09T14:06:27.113 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-03-09T14:06:27.113 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836494 2024-03-09T14:06:27.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:06:27.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836494 2024-03-09T14:06:27.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:06:27.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672966 2024-03-09T14:06:27.394 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:06:27.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:06:27.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672966 2024-03-09T14:06:27.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:06:27.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672966 2024-03-09T14:06:27.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-03-09T14:06:27.397 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672966 2024-03-09T14:06:27.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:06:27.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672966 -lt 42949672966 2024-03-09T14:06:27.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:06:27.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:06:27.676 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:06:28.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-03-09T14:06:28.046 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:06:28.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:06:28.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:06:28.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:06:28.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:06:28.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:06:28.046 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:06:28.314 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-03-09T14:06:28.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:06:28.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:06:28.315 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:06:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-03-09T14:06:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:06:28.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:06:28.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5772: TEST_periodic_scrub_replicated: create_pool psr_pool 1 1 2024-03-09T14:06:28.652 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create psr_pool 1 1 2024-03-09T14:06:29.026 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'psr_pool' created 2024-03-09T14:06:29.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T14:06:30.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5773: TEST_periodic_scrub_replicated: wait_for_clean 2024-03-09T14:06:30.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:06:30.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:06:30.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:06:30.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:06:30.040 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:06:30.040 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:06:30.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:06:30.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:06:30.041 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:06:30.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:06:30.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:06:30.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:06:30.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:06:30.143 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:06:30.144 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:06:30.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:06:30.463 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T14:06:30.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:06:30.463 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:06:30.463 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:06:30.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836496 2024-03-09T14:06:30.567 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836496 2024-03-09T14:06:30.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496' 2024-03-09T14:06:30.568 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:06:30.568 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:06:30.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672968 2024-03-09T14:06:30.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672968 2024-03-09T14:06:30.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836496 1-42949672968' 2024-03-09T14:06:30.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:06:30.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836496 2024-03-09T14:06:30.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:06:30.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:06:30.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836496 2024-03-09T14:06:30.682 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:06:30.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836496 2024-03-09T14:06:30.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-03-09T14:06:30.683 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836496 2024-03-09T14:06:30.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:06:30.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836494 -lt 21474836496 2024-03-09T14:06:30.952 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:06:31.953 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:06:31.953 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:06:32.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836496 -lt 21474836496 2024-03-09T14:06:32.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:06:32.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672968 2024-03-09T14:06:32.263 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:06:32.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:06:32.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672968 2024-03-09T14:06:32.265 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:06:32.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672968 2024-03-09T14:06:32.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-03-09T14:06:32.266 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672968 2024-03-09T14:06:32.267 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:06:32.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672968 -lt 42949672968 2024-03-09T14:06:32.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:06:32.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:06:32.542 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:06:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T14:06:32.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:06:32.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:06:32.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:06:32.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:06:32.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:06:32.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:06:32.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:06:33.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T14:06:33.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:06:33.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:06:33.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:06:33.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T14:06:33.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:06:33.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:06:33.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5775: TEST_periodic_scrub_replicated: local osd=0 2024-03-09T14:06:33.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5776: TEST_periodic_scrub_replicated: add_something td/osd-scrub-repair psr_pool POBJ scrub 2024-03-09T14:06:33.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2024-03-09T14:06:33.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2024-03-09T14:06:33.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2024-03-09T14:06:33.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=scrub 2024-03-09T14:06:33.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' scrub = noscrub ']' 2024-03-09T14:06:33.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:81: add_something: ceph osd unset noscrub 2024-03-09T14:06:33.906 INFO:tasks.workunit.client.0.smithi162.stderr:noscrub is unset 2024-03-09T14:06:33.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:82: add_something: ceph osd unset nodeep-scrub 2024-03-09T14:06:34.425 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is unset 2024-03-09T14:06:34.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2024-03-09T14:06:34.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2024-03-09T14:06:34.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2024-03-09T14:06:34.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: get_primary psr_pool POBJ 2024-03-09T14:06:34.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=psr_pool 2024-03-09T14:06:34.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=POBJ 2024-03-09T14:06:34.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map psr_pool POBJ 2024-03-09T14:06:34.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T14:06:34.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5777: TEST_periodic_scrub_replicated: local primary=0 2024-03-09T14:06:34.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: get_pg psr_pool POBJ 2024-03-09T14:06:34.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=psr_pool 2024-03-09T14:06:34.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=POBJ 2024-03-09T14:06:34.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map psr_pool POBJ 2024-03-09T14:06:34.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T14:06:35.258 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5778: TEST_periodic_scrub_replicated: local pg=2.0 2024-03-09T14:06:35.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5781: TEST_periodic_scrub_replicated: local payload=UVWXYZ 2024-03-09T14:06:35.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5782: TEST_periodic_scrub_replicated: echo UVWXYZ 2024-03-09T14:06:35.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5784: TEST_periodic_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T14:06:35.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-scrub-repair 2024-03-09T14:06:35.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-03-09T14:06:35.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-03-09T14:06:35.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-03-09T14:06:35.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T14:06:35.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2024-03-09T14:06:35.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-03-09T14:06:35.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-03-09T14:06:35.259 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-03-09T14:06:35.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T14:06:35.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:06:35.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:06:35.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:06:35.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:06:35.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:06:35.565 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:06:35.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T14:06:35.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:06:35.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:06:35.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:06:35.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:06:35.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:06:35.566 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 POBJ set-bytes td/osd-scrub-repair/CORRUPT 2024-03-09T14:06:37.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-09T14:06:37.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T14:06:37.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T14:06:37.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T14:06:37.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T14:06:37.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:06:37.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:06:37.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:06:37.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:06:37.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:06:37.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T14:06:37.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T14:06:37.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T14:06:37.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T14:06:37.496 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:06:37.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T14:06:37.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:06:37.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:06:37.496 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:06:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:06:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:06:37.497 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:06:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:06:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:06:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:06:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:06:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:06:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:06:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:06:37.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:06:37.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T14:06:37.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2024-03-09T14:06:37.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T14:06:37.498 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T14:06:37.498 INFO:tasks.workunit.client.0.smithi162.stderr:start osd.0 2024-03-09T14:06:37.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2024-03-09T14:06:37.499 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T14:06:37.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T14:06:37.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T14:06:37.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T14:06:37.499 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T14:06:37.526 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:06:37.526+0000 7f334bec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:06:37.528 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:06:37.529+0000 7f334bec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:06:37.530 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:06:37.530+0000 7f334bec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:06:37.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T14:06:37.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:06:37.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T14:06:37.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:06:37.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:06:37.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:37.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:06:37.782 INFO:tasks.workunit.client.0.smithi162.stderr:0 2024-03-09T14:06:37.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:37.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:06:38.046 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:06:38.047+0000 7f334bec6640 -1 Falling back to public interface 2024-03-09T14:06:38.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:39.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:39.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:39.059 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:06:39.059 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:06:39.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:39.060 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:06:39.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:40.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:40.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:40.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:06:40.343 INFO:tasks.workunit.client.0.smithi162.stderr:2 2024-03-09T14:06:40.343 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:40.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:06:40.425 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:06:40.426+0000 7f334bec6640 -1 osd.0 25 log_to_monitors true 2024-03-09T14:06:40.653 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:06:41.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:06:41.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:06:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:06:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:3 2024-03-09T14:06:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:06:41.655 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:06:42.112 INFO:tasks.workunit.client.0.smithi162.stderr:osd.0 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [5,25) [v2:127.0.0.1:6802/1659222505,v1:127.0.0.1:6803/1659222505] [v2:127.0.0.1:6804/1659222505,v1:127.0.0.1:6805/1659222505] exists,up 47219909-4497-48c1-aef2-920fb8a580ae 2024-03-09T14:06:42.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:06:42.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:06:42.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:06:42.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-03-09T14:06:42.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:06:42.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:06:42.112 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:06:42.112 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:06:42.113 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:06:42.113 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:06:42.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:06:42.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:06:42.113 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:06:42.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:06:42.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:06:42.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:06:42.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:06:42.203 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:06:42.204 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:06:42.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:06:42.505 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T14:06:42.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:06:42.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:06:42.505 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:06:42.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-03-09T14:06:42.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-03-09T14:06:42.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-03-09T14:06:42.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:06:42.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:06:42.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672972 2024-03-09T14:06:42.712 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672972 2024-03-09T14:06:42.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-42949672972' 2024-03-09T14:06:42.713 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:06:42.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-03-09T14:06:42.713 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:06:42.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:06:42.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-03-09T14:06:42.715 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:06:42.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-03-09T14:06:42.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-03-09T14:06:42.716 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.0 seq 124554051586 2024-03-09T14:06:42.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:06:42.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-03-09T14:06:42.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:06:43.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:06:43.987 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:06:44.268 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-03-09T14:06:44.269 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:06:44.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672972 2024-03-09T14:06:44.269 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:06:44.270 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:06:44.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672972 2024-03-09T14:06:44.271 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:06:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672972 2024-03-09T14:06:44.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-03-09T14:06:44.273 INFO:tasks.workunit.client.0.smithi162.stderr:waiting osd.1 seq 42949672972 2024-03-09T14:06:44.273 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:06:44.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672972 -lt 42949672972 2024-03-09T14:06:44.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:06:44.547 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:06:44.548 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:06:44.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-03-09T14:06:44.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:06:44.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:06:44.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:06:44.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:06:44.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:06:44.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:06:44.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:06:45.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-03-09T14:06:45.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:06:45.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:06:45.158 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:06:45.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-03-09T14:06:45.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:06:45.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:06:45.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5787: TEST_periodic_scrub_replicated: set -o pipefail 2024-03-09T14:06:45.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-09T14:06:45.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5788: TEST_periodic_scrub_replicated: jq . 2024-03-09T14:06:45.532 INFO:tasks.workunit.client.0.smithi162.stderr:No scrub information available for pg 2.0 2024-03-09T14:06:45.532 INFO:tasks.workunit.client.0.smithi162.stderr:error 2: (2) No such file or directory 2024-03-09T14:06:45.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5789: TEST_periodic_scrub_replicated: set +o pipefail 2024-03-09T14:06:45.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5791: TEST_periodic_scrub_replicated: pg_deep_scrub 2.0 2024-03-09T14:06:45.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1882: pg_deep_scrub: local pgid=2.0 2024-03-09T14:06:45.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-09T14:06:45.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:06:45.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T14:06:45.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:06:45.535 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-09T14:06:45.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1883: pg_deep_scrub: local last_scrub=2024-03-09T14:06:28.970043+0000 2024-03-09T14:06:45.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1884: pg_deep_scrub: ceph pg deep-scrub 2.0 2024-03-09T14:06:46.060 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0 on osd.0 to deep-scrub 2024-03-09T14:06:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1885: pg_deep_scrub: wait_for_scrub 2.0 2024-03-09T14:06:28.970043+0000 last_deep_scrub_stamp 2024-03-09T14:06:46.072 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T14:06:46.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T14:06:28.970043+0000 2024-03-09T14:06:46.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_deep_scrub_stamp 2024-03-09T14:06:46.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T14:06:46.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:06:46.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-09T14:06:46.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:06:46.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T14:06:46.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:06:46.073 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-09T14:06:46.340 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:06:28.970043+0000 '>' 2024-03-09T14:06:28.970043+0000 2024-03-09T14:06:46.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:06:47.341 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:06:47.342 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:06:47.342 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-09T14:06:47.342 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:06:47.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T14:06:47.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:06:47.343 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-09T14:06:47.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:06:28.970043+0000 '>' 2024-03-09T14:06:28.970043+0000 2024-03-09T14:06:47.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:06:48.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:06:48.677 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:06:48.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-09T14:06:48.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:06:48.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T14:06:48.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:06:48.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-09T14:06:48.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:06:28.970043+0000 '>' 2024-03-09T14:06:28.970043+0000 2024-03-09T14:06:48.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:06:49.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:06:49.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:06:49.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-09T14:06:49.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:06:49.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T14:06:49.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:06:49.960 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-09T14:06:50.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:06:28.970043+0000 '>' 2024-03-09T14:06:28.970043+0000 2024-03-09T14:06:50.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:06:51.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:06:51.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:06:51.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_deep_scrub_stamp 2024-03-09T14:06:51.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:06:51.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2024-03-09T14:06:51.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:06:51.248 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_deep_scrub_stamp' 2024-03-09T14:06:51.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:06:46.568166+0000 '>' 2024-03-09T14:06:28.970043+0000 2024-03-09T14:06:51.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T14:06:51.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-09T14:06:51.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: jq . 2024-03-09T14:06:51.532 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5794: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-09T14:06:51.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5796: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-09T14:06:51.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:06:51.558 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:06:51.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:06:51.829 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T14:06:51.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:06:51.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:06:51.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:06:51.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-03-09T14:06:51.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-03-09T14:06:51.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589' 2024-03-09T14:06:51.933 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:06:51.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:06:52.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672974 2024-03-09T14:06:52.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672974 2024-03-09T14:06:52.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-42949672974' 2024-03-09T14:06:52.038 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:06:52.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051589 2024-03-09T14:06:52.038 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:06:52.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:06:52.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051589 2024-03-09T14:06:52.040 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:06:52.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-03-09T14:06:52.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-03-09T14:06:52.041 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 124554051589 2024-03-09T14:06:52.042 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:06:52.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-03-09T14:06:52.315 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:06:53.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:06:53.316 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:06:53.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051589 2024-03-09T14:06:53.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:06:53.600 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672974 2024-03-09T14:06:53.601 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:06:53.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:06:53.602 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672974 2024-03-09T14:06:53.606 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:06:53.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672974 2024-03-09T14:06:53.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-03-09T14:06:53.606 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672974 2024-03-09T14:06:53.607 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:06:53.878 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672975 -lt 42949672974 2024-03-09T14:06:53.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: get_last_scrub_stamp 2.0 2024-03-09T14:06:53.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:06:53.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:06:53.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:06:53.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:06:54.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5797: TEST_periodic_scrub_replicated: local last_scrub=2024-03-09T14:06:46.568166+0000 2024-03-09T14:06:54.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5799: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-03-09T14:06:54.243 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T14:06:54.243 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2024-03-09T14:06:54.243 INFO:tasks.workunit.client.0.smithi162.stdout: "must": false, 2024-03-09T14:06:54.243 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "2024-03-02T14:05:14.243755+0000" 2024-03-09T14:06:54.243 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T14:06:54.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5801: TEST_periodic_scrub_replicated: wait_for_scrub 2.0 2024-03-09T14:06:46.568166+0000 2024-03-09T14:06:54.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T14:06:54.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T14:06:46.568166+0000 2024-03-09T14:06:54.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T14:06:54.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T14:06:54.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:06:54.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T14:06:54.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:06:54.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:06:54.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:06:54.253 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:06:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:06:46.568166+0000 '>' 2024-03-09T14:06:46.568166+0000 2024-03-09T14:06:54.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:06:55.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:06:55.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:06:55.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T14:06:55.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:06:55.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:06:55.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:06:55.526 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:06:55.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:06:46.568166+0000 '>' 2024-03-09T14:06:46.568166+0000 2024-03-09T14:06:55.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:06:56.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:06:56.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:06:56.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T14:06:56.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:06:56.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:06:56.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:06:56.808 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:06:57.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:06:46.568166+0000 '>' 2024-03-09T14:06:46.568166+0000 2024-03-09T14:06:57.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:06:58.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:06:58.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:06:58.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T14:06:58.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:06:58.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:06:58.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:06:58.090 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:06:58.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:06:54.599449+0000 '>' 2024-03-09T14:06:46.568166+0000 2024-03-09T14:06:58.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T14:06:58.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5804: TEST_periodic_scrub_replicated: grep -q 'Deep scrub errors, upgrading scrub to deep-scrub' td/osd-scrub-repair/osd.0.log 2024-03-09T14:06:58.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-09T14:06:58.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: jq . 2024-03-09T14:06:58.371 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5807: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-09T14:06:58.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5810: TEST_periodic_scrub_replicated: ceph osd set nodeep-scrub 2024-03-09T14:06:58.771 INFO:tasks.workunit.client.0.smithi162.stderr:nodeep-scrub is set 2024-03-09T14:06:58.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5812: TEST_periodic_scrub_replicated: ceph tell osd.0 get_latest_osdmap 2024-03-09T14:06:58.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5813: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-09T14:06:58.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:06:58.887 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:06:59.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:06:59.161 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T14:06:59.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:06:59.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:06:59.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:06:59.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051591 2024-03-09T14:06:59.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051591 2024-03-09T14:06:59.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591' 2024-03-09T14:06:59.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:06:59.294 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:06:59.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672977 2024-03-09T14:06:59.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672977 2024-03-09T14:06:59.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-42949672977' 2024-03-09T14:06:59.398 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:06:59.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051591 2024-03-09T14:06:59.398 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:06:59.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:06:59.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051591 2024-03-09T14:06:59.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:06:59.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051591 2024-03-09T14:06:59.401 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 124554051591 2024-03-09T14:06:59.402 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2024-03-09T14:06:59.402 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:06:59.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051590 -lt 124554051591 2024-03-09T14:06:59.685 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:07:00.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:07:00.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:07:00.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051590 -lt 124554051591 2024-03-09T14:07:00.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:07:01.971 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 299 -eq 0 ']' 2024-03-09T14:07:01.971 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:07:02.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051592 -lt 124554051591 2024-03-09T14:07:02.245 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:07:02.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672977 2024-03-09T14:07:02.245 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:07:02.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:07:02.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672977 2024-03-09T14:07:02.247 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:07:02.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672977 2024-03-09T14:07:02.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-03-09T14:07:02.249 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672977 2024-03-09T14:07:02.249 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:07:02.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672977 -lt 42949672977 2024-03-09T14:07:02.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5814: TEST_periodic_scrub_replicated: sleep 5 2024-03-09T14:07:07.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5817: TEST_periodic_scrub_replicated: ceph tell 2.0 schedule-scrub 2024-03-09T14:07:07.632 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T14:07:07.632 INFO:tasks.workunit.client.0.smithi162.stdout: "deep": false, 2024-03-09T14:07:07.632 INFO:tasks.workunit.client.0.smithi162.stdout: "must": false, 2024-03-09T14:07:07.632 INFO:tasks.workunit.client.0.smithi162.stdout: "stamp": "2024-03-02T14:05:27.632714+0000" 2024-03-09T14:07:07.632 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T14:07:07.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5820: TEST_periodic_scrub_replicated: local found=false 2024-03-09T14:07:07.642 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: seq 14 -1 0 2024-03-09T14:07:07.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5821: TEST_periodic_scrub_replicated: for i in $(seq 14 -1 0) 2024-03-09T14:07:07.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5823: TEST_periodic_scrub_replicated: sleep 1 2024-03-09T14:07:08.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: grep -q 'Regular scrub skipped due to deep-scrub errors and nodeep-scrub set' td/osd-scrub-repair/osd.0.log 2024-03-09T14:07:08.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: found=true 2024-03-09T14:07:08.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5824: TEST_periodic_scrub_replicated: break 2024-03-09T14:07:08.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5827: TEST_periodic_scrub_replicated: test true = true 2024-03-09T14:07:08.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-09T14:07:08.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: jq . 2024-03-09T14:07:08.647 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5830: TEST_periodic_scrub_replicated: grep -q POBJ 2024-03-09T14:07:08.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5832: TEST_periodic_scrub_replicated: flush_pg_stats 2024-03-09T14:07:08.678 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:07:08.678 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:07:08.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:07:08.954 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T14:07:08.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:07:08.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:07:08.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:07:09.057 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051594 2024-03-09T14:07:09.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051594 2024-03-09T14:07:09.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051594' 2024-03-09T14:07:09.058 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:07:09.058 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:07:09.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-03-09T14:07:09.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-03-09T14:07:09.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051594 1-42949672980' 2024-03-09T14:07:09.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:07:09.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051594 2024-03-09T14:07:09.161 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:07:09.161 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:07:09.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051594 2024-03-09T14:07:09.162 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:07:09.163 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051594 2024-03-09T14:07:09.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051594' 2024-03-09T14:07:09.164 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 124554051594 2024-03-09T14:07:09.164 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:07:09.436 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051594 -lt 124554051594 2024-03-09T14:07:09.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:07:09.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-03-09T14:07:09.437 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:07:09.438 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:07:09.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-03-09T14:07:09.439 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:07:09.440 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-03-09T14:07:09.440 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672980 2024-03-09T14:07:09.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-03-09T14:07:09.441 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:07:09.715 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-03-09T14:07:09.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5834: TEST_periodic_scrub_replicated: pg_schedule_scrub 2.0 2024-03-09T14:07:09.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: pg_schedule_scrub: local pgid=2.0 2024-03-09T14:07:09.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: get_last_scrub_stamp 2.0 2024-03-09T14:07:09.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:07:09.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:07:09.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:07:09.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:07:09.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: pg_schedule_scrub: local last_scrub=2024-03-02T14:05:27.632714+0000 2024-03-09T14:07:09.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: pg_schedule_scrub: ceph pg scrub 2.0 2024-03-09T14:07:10.242 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 2.0 on osd.0 to scrub 2024-03-09T14:07:10.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: pg_schedule_scrub: wait_for_scrub 2.0 2024-03-02T14:05:27.632714+0000 2024-03-09T14:07:10.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=2.0 2024-03-09T14:07:10.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-02T14:05:27.632714+0000 2024-03-09T14:07:10.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T14:07:10.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T14:07:10.255 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:07:10.255 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T14:07:10.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:07:10.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:07:10.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:07:10.256 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:07:10.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-02T14:05:27.632714+0000 '>' 2024-03-02T14:05:27.632714+0000 2024-03-09T14:07:10.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:07:11.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:07:11.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:07:11.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2024-03-09T14:07:11.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=2.0 2024-03-09T14:07:11.527 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:07:11.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:07:11.528 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2024-03-09T14:07:11.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:07:10.365626+0000 '>' 2024-03-02T14:05:27.632714+0000 2024-03-09T14:07:11.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T14:07:11.799 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5835: TEST_periodic_scrub_replicated: grep -q 'Regular scrub request, deep-scrub details will be lost' td/osd-scrub-repair/osd.0.log 2024-03-09T14:07:11.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: rados list-inconsistent-obj 2.0 2024-03-09T14:07:11.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: jq . 2024-03-09T14:07:11.801 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5838: TEST_periodic_scrub_replicated: grep -qv POBJ 2024-03-09T14:07:11.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T14:07:11.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T14:07:11.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T14:07:11.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T14:07:11.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:07:11.828 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:07:11.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:07:11.828 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:07:11.829 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:07:11.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:07:11.936 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T14:07:11.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T14:07:11.938 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T14:07:11.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T14:07:11.939 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T14:07:11.939 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T14:07:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:07:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T14:07:11.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T14:07:11.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:07:11.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T14:07:11.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T14:07:11.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T14:07:11.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T14:07:11.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:07:11.955 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:07:11.955 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T14:07:11.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T14:07:11.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T14:07:11.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T14:07:11.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T14:07:11.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T14:07:11.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T14:07:11.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T14:07:11.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T14:07:11.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T14:07:11.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:07:11.958 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:07:11.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:07:11.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:07:11.958 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:07:11.959 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:07:11.959 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T14:07:11.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T14:07:11.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T14:07:11.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T14:07:11.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T14:07:11.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T14:07:11.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:07:11.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T14:07:11.963 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T14:07:11.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:07:11.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T14:07:11.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T14:07:11.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T14:07:11.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T14:07:11.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:07:11.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:07:11.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T14:07:11.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T14:07:11.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T14:07:11.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T14:07:11.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T14:07:11.969 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:07:11.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:07:11.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T14:07:11.972 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T14:07:11.972 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_repair_stats: local dir=td/osd-scrub-repair 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:655: TEST_repair_stats: local poolname=testpool 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_repair_stats: local OSDS=2 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_repair_stats: local OBJS=30 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_repair_stats: local REPAIRS=20 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:662: TEST_repair_stats: run_mon td/osd-scrub-repair a 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T14:07:11.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T14:07:11.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T14:07:12.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T14:07:12.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:07:12.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:07:12.431 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:07:12.431 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:07:12.431 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:07:12.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:07:12.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T14:07:12.614 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T14:07:12.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T14:07:12.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T14:07:12.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T14:07:12.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T14:07:12.615 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T14:07:12.616 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T14:07:12.616 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T14:07:12.617 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T14:07:12.617 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T14:07:12.617 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:07:12.617 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:07:12.617 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T14:07:12.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T14:07:12.617 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T14:07:12.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T14:07:12.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T14:07:12.699 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T14:07:12.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T14:07:12.700 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T14:07:12.700 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T14:07:12.700 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T14:07:12.700 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T14:07:12.700 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T14:07:12.700 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:07:12.700 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:07:12.700 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T14:07:12.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T14:07:12.701 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T14:07:12.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2024-03-09T14:07:12.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T14:07:12.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T14:07:12.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T14:07:12.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T14:07:12.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T14:07:12.779 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T14:07:13.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T14:07:13.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:07:13.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:07:13.591 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:07:13.591 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:07:13.591 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:07:13.591 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:07:13.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T14:07:13.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T14:07:13.619 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:665: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T14:07:13.619 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: expr 2 - 1 2024-03-09T14:07:13.621 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: seq 0 1 2024-03-09T14:07:13.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T14:07:13.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:07:13.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T14:07:13.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T14:07:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T14:07:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T14:07:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:07:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:07:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:07:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:07:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:07:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T14:07:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T14:07:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T14:07:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T14:07:13.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:07:13.626 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:07:13.627 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:07:13.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:07:13.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:07:13.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T14:07:13.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T14:07:13.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T14:07:13.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T14:07:13.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ee0bac3-05f8-4687-b8f0-e293c4c3fade 2024-03-09T14:07:13.628 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6ee0bac3-05f8-4687-b8f0-e293c4c3fade' 2024-03-09T14:07:13.628 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 6ee0bac3-05f8-4687-b8f0-e293c4c3fade 2024-03-09T14:07:13.628 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T14:07:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARbexlTAtcJhAAzySsODrHbKV8B2rUswL8Wg== 2024-03-09T14:07:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARbexlTAtcJhAAzySsODrHbKV8B2rUswL8Wg=="}' 2024-03-09T14:07:13.644 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ee0bac3-05f8-4687-b8f0-e293c4c3fade -i td/osd-scrub-repair/0/new.json 2024-03-09T14:07:13.847 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:07:13.857 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T14:07:13.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQARbexlTAtcJhAAzySsODrHbKV8B2rUswL8Wg== --osd-uuid 6ee0bac3-05f8-4687-b8f0-e293c4c3fade 2024-03-09T14:07:13.888 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:13.888+0000 7fb8aa80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:07:13.898 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:13.898+0000 7fb8aa80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:07:13.899 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:13.899+0000 7fb8aa80d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:07:13.899 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:13.900+0000 7fb8aa80d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:07:13.899 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:13.900+0000 7fb8aa80d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:07:13.899 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:13.900+0000 7fb8aa80d640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:07:13.990 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:13.990+0000 7fb8aa80d640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T14:07:19.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T14:07:19.558 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T14:07:19.559 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T14:07:19.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T14:07:19.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T14:07:19.916 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T14:07:19.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T14:07:19.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T14:07:19.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:07:19.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T14:07:19.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T14:07:19.944 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:19.944+0000 7f1ec0ddf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:07:19.946 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:19.946+0000 7f1ec0ddf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:07:19.947 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:19.947+0000 7f1ec0ddf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:07:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T14:07:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:07:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T14:07:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:07:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:07:20.191 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:20.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:07:20.192 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:07:20.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:20.192 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:20.462 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:20.473 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:20.474+0000 7f1ec0ddf640 -1 Falling back to public interface 2024-03-09T14:07:21.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:21.464 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:21.464 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:07:21.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:07:21.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:21.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:21.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:22.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:22.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:22.747 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:07:22.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:07:22.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:22.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:22.828 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:22.828+0000 7f1ec0ddf640 -1 osd.0 0 log_to_monitors true 2024-03-09T14:07:23.094 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:24.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:24.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:24.096 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:07:24.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:07:24.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:24.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:24.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:25.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:25.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:25.392 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T14:07:25.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T14:07:25.392 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:25.393 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:25.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:26.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:26.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:26.769 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T14:07:26.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T14:07:26.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:26.769 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:27.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:28.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:28.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:28.053 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T14:07:28.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T14:07:28.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:28.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:28.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:29.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:29.356 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:29.356 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T14:07:29.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T14:07:29.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:29.357 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:29.722 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:30.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:30.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:30.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T14:07:30.724 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T14:07:30.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:30.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:31.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:32.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:32.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:32.010 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T14:07:32.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T14:07:32.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:32.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:32.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:33.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:33.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:33.294 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T14:07:33.294 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T14:07:33.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:33.295 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:33.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:34.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:34.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:34.581 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T14:07:34.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T14:07:34.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:34.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:35.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:36.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:36.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:36.089 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T14:07:36.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T14:07:36.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:36.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:36.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:37.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:37.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:37.610 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T14:07:37.610 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T14:07:37.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:37.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:38.431 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:39.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:39.432 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:39.432 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T14:07:39.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T14:07:39.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:39.433 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:39.975 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:40.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:40.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:40.977 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T14:07:40.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T14:07:40.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:40.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:41.557 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:42.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:42.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:42.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T14:07:42.559 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T14:07:42.559 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:42.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:43.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:44.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:44.200 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:44.201 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T14:07:44.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T14:07:44.201 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:44.202 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:44.606 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:45.607 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:45.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:45.608 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T14:07:45.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T14:07:45.608 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:45.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:07:45.978 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/680139562,v1:127.0.0.1:6803/680139562] [v2:127.0.0.1:6804/680139562,v1:127.0.0.1:6805/680139562] exists,up 6ee0bac3-05f8-4687-b8f0-e293c4c3fade 2024-03-09T14:07:45.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:07:45.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:07:45.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:07:45.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:666: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T14:07:45.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:07:45.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T14:07:45.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T14:07:45.984 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T14:07:45.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T14:07:45.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:07:45.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:07:45.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:07:45.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:07:45.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:07:45.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T14:07:45.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T14:07:45.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T14:07:45.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T14:07:45.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:07:45.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T14:07:45.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:07:45.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:07:45.986 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:07:45.986 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:07:45.986 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:07:45.986 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:07:45.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:07:45.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:07:45.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T14:07:45.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T14:07:45.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:07:45.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:07:45.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:07:45.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:07:45.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:07:45.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:07:45.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T14:07:45.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T14:07:45.988 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T14:07:45.988 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T14:07:45.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=96ede09d-584c-4739-90d5-7f0b9af6af5e 2024-03-09T14:07:45.989 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 96ede09d-584c-4739-90d5-7f0b9af6af5e 2024-03-09T14:07:45.989 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 96ede09d-584c-4739-90d5-7f0b9af6af5e' 2024-03-09T14:07:45.990 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T14:07:46.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxbexlZYqaOxAA20WWh0YBe9Ra5mvnS7RHmQ== 2024-03-09T14:07:46.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxbexlZYqaOxAA20WWh0YBe9Ra5mvnS7RHmQ=="}' 2024-03-09T14:07:46.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 96ede09d-584c-4739-90d5-7f0b9af6af5e -i td/osd-scrub-repair/1/new.json 2024-03-09T14:07:46.326 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:07:46.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T14:07:46.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAxbexlZYqaOxAA20WWh0YBe9Ra5mvnS7RHmQ== --osd-uuid 96ede09d-584c-4739-90d5-7f0b9af6af5e 2024-03-09T14:07:46.368 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:46.367+0000 7fc5e8a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:07:46.419 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:46.418+0000 7fc5e8a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:07:46.419 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:46.419+0000 7fc5e8a13640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:07:46.420 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:46.419+0000 7fc5e8a13640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:07:46.420 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:46.419+0000 7fc5e8a13640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:07:46.420 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:46.419+0000 7fc5e8a13640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:07:46.494 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:46.493+0000 7fc5e8a13640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T14:07:52.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T14:07:52.600 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T14:07:52.601 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T14:07:52.601 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T14:07:52.602 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T14:07:52.953 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T14:07:52.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T14:07:52.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:07:52.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T14:07:52.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T14:07:52.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T14:07:52.981 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:52.980+0000 7f356520f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:07:52.983 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:52.982+0000 7f356520f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:07:52.984 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:52.983+0000 7f356520f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:07:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T14:07:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:07:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T14:07:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:07:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:07:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:53.229 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:07:53.230 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:07:53.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:53.230 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:07:53.501 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:53.501+0000 7f356520f640 -1 Falling back to public interface 2024-03-09T14:07:53.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:54.503 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:07:54.504 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:07:54.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:54.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:07:54.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:55.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:55.784 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:55.784 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:07:55.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:07:55.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:55.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:07:56.009 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:07:56.008+0000 7f356520f640 -1 osd.1 0 log_to_monitors true 2024-03-09T14:07:56.225 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:57.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:57.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:57.227 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:07:57.227 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:07:57.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:57.228 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:07:57.500 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:58.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:58.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:58.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T14:07:58.502 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T14:07:58.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:58.502 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:07:58.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:07:59.850 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:07:59.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:07:59.851 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T14:07:59.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T14:07:59.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:07:59.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:00.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:01.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:01.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:01.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T14:08:01.137 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T14:08:01.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:01.137 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:01.419 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:02.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T14:08:02.421 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T14:08:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:02.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:02.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:03.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:03.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:03.706 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T14:08:03.707 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T14:08:03.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:03.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:04.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:05.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:05.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:05.106 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T14:08:05.106 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T14:08:05.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:05.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:05.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:06.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:06.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:06.384 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T14:08:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T14:08:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:06.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:06.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:07.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:07.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:07.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T14:08:07.669 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T14:08:07.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:07.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:07.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:08.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:08.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:08.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T14:08:08.966 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T14:08:08.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:08.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:09.336 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:10.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:10.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:10.338 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T14:08:10.338 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T14:08:10.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:10.339 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:10.668 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:11.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:11.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:11.669 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T14:08:11.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T14:08:11.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:11.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:12.020 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:13.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:13.021 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:13.021 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T14:08:13.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T14:08:13.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:13.022 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:13.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:14.724 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:14.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:14.725 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T14:08:14.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T14:08:14.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:14.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:15.241 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:16.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:16.246 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:16.246 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T14:08:16.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T14:08:16.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:16.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:16.915 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:17.916 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:17.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:17.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T14:08:17.917 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T14:08:17.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:17.917 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:18.621 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:19.623 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T14:08:19.623 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T14:08:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:19.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:19.926 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1015977444,v1:127.0.0.1:6811/1015977444] [v2:127.0.0.1:6812/1015977444,v1:127.0.0.1:6813/1015977444] exists,up 96ede09d-584c-4739-90d5-7f0b9af6af5e 2024-03-09T14:08:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:08:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:08:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:08:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:670: TEST_repair_stats: create_pool testpool 1 1 2024-03-09T14:08:19.927 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 2024-03-09T14:08:20.348 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'testpool' created 2024-03-09T14:08:20.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T14:08:21.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_repair_stats: ceph osd pool set testpool size 2 2024-03-09T14:08:21.743 INFO:tasks.workunit.client.0.smithi162.stderr:set pool 1 size to 2 2024-03-09T14:08:21.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_repair_stats: wait_for_clean 2024-03-09T14:08:21.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:08:21.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:08:21.755 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:08:21.755 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:08:21.756 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:08:21.756 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:08:21.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:08:21.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:08:21.756 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:08:21.847 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:08:21.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:08:21.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:08:21.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:08:21.848 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:08:21.848 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:08:22.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:08:22.293 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T14:08:22.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:08:22.293 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:08:22.293 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:08:22.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836491 2024-03-09T14:08:22.425 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836491 2024-03-09T14:08:22.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491' 2024-03-09T14:08:22.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:08:22.426 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:08:22.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672967 2024-03-09T14:08:22.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672967 2024-03-09T14:08:22.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836491 1-42949672967' 2024-03-09T14:08:22.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:08:22.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836491 2024-03-09T14:08:22.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:08:22.579 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:08:22.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836491 2024-03-09T14:08:22.580 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:08:22.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836491 2024-03-09T14:08:22.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-03-09T14:08:22.582 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836491 2024-03-09T14:08:22.582 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:08:22.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836490 -lt 21474836491 2024-03-09T14:08:22.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:08:23.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:08:23.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:08:24.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836491 -lt 21474836491 2024-03-09T14:08:24.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:08:24.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672967 2024-03-09T14:08:24.174 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:08:24.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:08:24.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672967 2024-03-09T14:08:24.176 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:08:24.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672967 2024-03-09T14:08:24.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-03-09T14:08:24.177 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672967 2024-03-09T14:08:24.177 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:08:24.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672967 -lt 42949672967 2024-03-09T14:08:24.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:08:24.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:08:24.450 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:08:24.791 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T14:08:24.792 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:08:24.792 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:08:24.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:08:24.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:08:24.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:08:24.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:08:24.793 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:08:25.062 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T14:08:25.062 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:08:25.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:08:25.063 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:08:25.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T14:08:25.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:08:25.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:08:25.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:675: TEST_repair_stats: local payload=ABCDEF 2024-03-09T14:08:25.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_repair_stats: echo ABCDEF 2024-03-09T14:08:25.400 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: seq 1 30 2024-03-09T14:08:25.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:25.400 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:25.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:25.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:25.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:25.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:25.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:25.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:26.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:26.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:26.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:26.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:26.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:26.300 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:26.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:26.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:26.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:26.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:26.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:26.676 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:26.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:26.810 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:26.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:26.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:27.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:27.070 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:27.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:27.195 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:27.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:27.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:27.446 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:27.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:27.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:27.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:27.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:27.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:27.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:27.899 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:28.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:28.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:28.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:28.150 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:28.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:28.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:28.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:28.401 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:28.526 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:28.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:28.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:28.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:28.785 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:28.786 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:28.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:28.920 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:29.053 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:29.054 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:29.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:29.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:29.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_repair_stats: for i in $(seq 1 $OBJS) 2024-03-09T14:08:29.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:679: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:08:29.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: get_not_primary testpool obj1 2024-03-09T14:08:29.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T14:08:29.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-09T14:08:29.442 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-09T14:08:29.442 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T14:08:29.442 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T14:08:29.442 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T14:08:29.442 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T14:08:29.725 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T14:08:29.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-09T14:08:29.726 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T14:08:30.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:684: TEST_repair_stats: local other=0 2024-03-09T14:08:30.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: get_pg testpool obj1 2024-03-09T14:08:30.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T14:08:30.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-09T14:08:30.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-09T14:08:30.003 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T14:08:30.287 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_repair_stats: local pgid=1.0 2024-03-09T14:08:30.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: get_primary testpool obj1 2024-03-09T14:08:30.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T14:08:30.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T14:08:30.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T14:08:30.288 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T14:08:30.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:686: TEST_repair_stats: local primary=1 2024-03-09T14:08:30.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:688: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T14:08:30.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:08:30.563 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:08:30.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:08:30.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:08:30.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:08:30.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:08:30.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:689: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T14:08:30.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:08:30.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:08:30.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:08:30.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:08:30.871 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:08:31.177 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:08:31.178 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: seq 1 20 2024-03-09T14:08:31.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:31.179 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 1 % 2 2024-03-09T14:08:31.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T14:08:31.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-03-09T14:08:31.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:31.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:31.181 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:08:31.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:31.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:08:31.182 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-03-09T14:08:32.424 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:ff7b1f36:::obj1:head# 2024-03-09T14:08:32.941 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:32.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 2 % 2 2024-03-09T14:08:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T14:08:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-09T14:08:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:32.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:32.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:08:32.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:32.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:08:32.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-09T14:08:34.187 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:104778fc:::obj2:head# 2024-03-09T14:08:34.702 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:34.702 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 3 % 2 2024-03-09T14:08:34.704 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T14:08:34.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-03-09T14:08:34.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:34.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:34.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:08:34.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:34.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:08:34.705 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-03-09T14:08:35.488 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:8dd16f86:::obj3:head# 2024-03-09T14:08:36.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:36.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 4 % 2 2024-03-09T14:08:36.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T14:08:36.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-03-09T14:08:36.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:36.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:36.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:08:36.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:36.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:08:36.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-03-09T14:08:36.749 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:0ee9ae15:::obj4:head# 2024-03-09T14:08:37.264 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:37.264 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 5 % 2 2024-03-09T14:08:37.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T14:08:37.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-03-09T14:08:37.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:37.266 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:37.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:08:37.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:37.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:08:37.267 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-03-09T14:08:38.060 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:c52c9666:::obj5:head# 2024-03-09T14:08:38.578 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:38.578 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 6 % 2 2024-03-09T14:08:38.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T14:08:38.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-03-09T14:08:38.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:38.580 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:38.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:08:38.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:38.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:08:38.581 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-03-09T14:08:39.253 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:233a42c1:::obj6:head# 2024-03-09T14:08:39.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:39.775 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 7 % 2 2024-03-09T14:08:39.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T14:08:39.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-03-09T14:08:39.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:39.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:39.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:08:39.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:39.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:08:39.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-03-09T14:08:40.439 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:9a09113a:::obj7:head# 2024-03-09T14:08:40.954 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:40.954 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 8 % 2 2024-03-09T14:08:40.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T14:08:40.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-03-09T14:08:40.956 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:40.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:40.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:08:40.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:40.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:08:40.957 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-03-09T14:08:41.607 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:a557efb1:::obj8:head# 2024-03-09T14:08:42.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:42.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 9 % 2 2024-03-09T14:08:42.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T14:08:42.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-03-09T14:08:42.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:42.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:42.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:08:42.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:42.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:08:42.127 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-03-09T14:08:42.784 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:fe351e27:::obj9:head# 2024-03-09T14:08:43.298 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:43.299 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 10 % 2 2024-03-09T14:08:43.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T14:08:43.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-03-09T14:08:43.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:43.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:43.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:08:43.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:43.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:08:43.301 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-03-09T14:08:44.178 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:d1337354:::obj10:head# 2024-03-09T14:08:44.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:44.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 11 % 2 2024-03-09T14:08:44.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T14:08:44.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-03-09T14:08:44.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:44.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:44.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:08:44.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:44.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:08:44.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-03-09T14:08:45.354 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:eeae2a94:::obj11:head# 2024-03-09T14:08:46.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:46.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 12 % 2 2024-03-09T14:08:46.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T14:08:46.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-03-09T14:08:46.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:46.125 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:46.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:08:46.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:46.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:08:46.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-03-09T14:08:46.774 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:af5f95cb:::obj12:head# 2024-03-09T14:08:47.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:47.540 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 13 % 2 2024-03-09T14:08:47.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T14:08:47.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-03-09T14:08:47.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:47.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:47.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:08:47.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:47.542 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:08:47.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-03-09T14:08:48.177 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:a61ad63e:::obj13:head# 2024-03-09T14:08:48.695 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:48.695 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 14 % 2 2024-03-09T14:08:48.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T14:08:48.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-03-09T14:08:48.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:48.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:48.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:08:48.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:48.697 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:08:48.698 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-03-09T14:08:49.362 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:e0a44829:::obj14:head# 2024-03-09T14:08:49.879 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:49.879 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 15 % 2 2024-03-09T14:08:49.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T14:08:49.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-03-09T14:08:49.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:49.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:49.881 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:08:49.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:49.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:08:49.882 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-03-09T14:08:50.481 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:ab946124:::obj15:head# 2024-03-09T14:08:50.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:50.999 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 16 % 2 2024-03-09T14:08:51.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T14:08:51.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-03-09T14:08:51.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:51.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:51.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:08:51.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:51.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:08:51.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-03-09T14:08:51.698 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:461f8b5e:::obj16:head# 2024-03-09T14:08:52.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:52.212 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 17 % 2 2024-03-09T14:08:52.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T14:08:52.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-03-09T14:08:52.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:52.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:52.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:08:52.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:52.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:08:52.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-03-09T14:08:52.850 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:24fc6e92:::obj17:head# 2024-03-09T14:08:53.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:53.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 18 % 2 2024-03-09T14:08:53.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T14:08:53.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-03-09T14:08:53.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:53.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:53.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:08:53.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:53.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:08:53.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-03-09T14:08:54.119 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:b9836a99:::obj18:head# 2024-03-09T14:08:54.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:54.634 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 19 % 2 2024-03-09T14:08:54.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=1 2024-03-09T14:08:54.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-03-09T14:08:54.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:54.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:54.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:08:54.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:54.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:08:54.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-03-09T14:08:55.270 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:00b31b6c:::obj19:head# 2024-03-09T14:08:55.795 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:690: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2024-03-09T14:08:55.795 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: expr 20 % 2 2024-03-09T14:08:55.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:693: TEST_repair_stats: OSD=0 2024-03-09T14:08:55.797 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-03-09T14:08:55.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:08:55.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:08:55.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:08:55.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:08:55.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:08:55.798 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-03-09T14:08:56.447 INFO:tasks.workunit.client.0.smithi162.stdout:remove #1:24b5611b:::obj20:head# 2024-03-09T14:08:56.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:08:56.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T14:08:56.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T14:08:56.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T14:08:56.965 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T14:08:56.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:08:56.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:08:56.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:08:56.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:08:56.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:08:56.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T14:08:56.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T14:08:56.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T14:08:56.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T14:08:56.966 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:08:56.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T14:08:56.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:08:56.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:08:56.967 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:08:56.967 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:08:56.967 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:08:56.967 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:08:56.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:08:56.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:08:56.967 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:08:56.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:08:56.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:08:56.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:08:56.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:08:56.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:08:56.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T14:08:56.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T14:08:56.968 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T14:08:56.968 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T14:08:56.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T14:08:56.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:08:56.970 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T14:08:56.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T14:08:56.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T14:08:56.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T14:08:56.970 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T14:08:56.997 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:08:56.997+0000 7f16098b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:08:56.998 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:08:56.999+0000 7f16098b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:08:57.000 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:08:57.000+0000 7f16098b5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:08:57.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T14:08:57.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:08:57.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T14:08:57.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:08:57.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:08:57.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:57.253 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:08:57.254 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:08:57.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:57.254 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:57.517 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:08:57.518+0000 7f16098b5640 -1 Falling back to public interface 2024-03-09T14:08:57.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:58.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:58.524 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:58.524 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:08:58.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:08:58.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:58.525 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:58.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:08:59.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:08:59.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:08:59.806 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:08:59.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:08:59.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:08:59.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:08:59.993 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:08:59.993+0000 7f16098b5640 -1 osd.1 17 log_to_monitors true 2024-03-09T14:09:00.167 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:01.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:01.168 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:01.169 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:09:01.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:09:01.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:01.169 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:09:01.444 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 20 up_thru 20 down_at 18 last_clean_interval [10,17) [v2:127.0.0.1:6802/3981357295,v1:127.0.0.1:6803/3981357295] [v2:127.0.0.1:6804/3981357295,v1:127.0.0.1:6805/3981357295] exists,up 96ede09d-584c-4739-90d5-7f0b9af6af5e 2024-03-09T14:09:01.447 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:697: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T14:09:01.448 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:09:01.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:09:01.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:09:01.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:09:01.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:09:01.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T14:09:01.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T14:09:01.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T14:09:01.450 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T14:09:01.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T14:09:01.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:09:01.451 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T14:09:01.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T14:09:01.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T14:09:01.451 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T14:09:01.452 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T14:09:01.474 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:01.474+0000 7fc42fa22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:09:01.476 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:01.476+0000 7fc42fa22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:09:01.477 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:01.477+0000 7fc42fa22640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:09:01.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T14:09:01.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:09:01.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T14:09:01.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:09:01.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:09:01.727 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:01.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:09:01.728 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:09:01.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:01.728 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:02.001 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:02.001 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:02.002+0000 7fc42fa22640 -1 Falling back to public interface 2024-03-09T14:09:03.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:03.003 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:03.004 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:09:03.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:09:03.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:03.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:03.275 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:04.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:04.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:04.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:09:04.277 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:09:04.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:04.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:04.277 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:04.277+0000 7fc42fa22640 -1 osd.0 16 log_to_monitors true 2024-03-09T14:09:04.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:05.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:05.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:05.640 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:09:05.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:09:05.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:05.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:05.920 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/1471687202,v1:127.0.0.1:6811/1471687202] [v2:127.0.0.1:6812/1471687202,v1:127.0.0.1:6813/1471687202] exists,up 6ee0bac3-05f8-4687-b8f0-e293c4c3fade 2024-03-09T14:09:05.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:09:05.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:09:05.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:09:05.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_repair_stats: wait_for_clean 2024-03-09T14:09:05.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:09:05.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:09:05.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:09:05.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:09:05.922 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:09:05.922 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:09:05.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:09:05.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:09:05.922 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:09:06.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:09:06.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:09:06.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:09:06.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:09:06.013 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:09:06.013 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:09:06.284 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:09:06.284 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T14:09:06.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:09:06.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:09:06.285 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:09:06.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247810 2024-03-09T14:09:06.390 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247810 2024-03-09T14:09:06.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810' 2024-03-09T14:09:06.391 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:09:06.391 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:09:06.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345923 2024-03-09T14:09:06.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345923 2024-03-09T14:09:06.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247810 1-85899345923' 2024-03-09T14:09:06.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:09:06.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247810 2024-03-09T14:09:06.495 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:09:06.495 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:09:06.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247810 2024-03-09T14:09:06.496 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:09:06.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247810 2024-03-09T14:09:06.497 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2024-03-09T14:09:06.498 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 98784247810 2024-03-09T14:09:06.498 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:09:06.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 98784247810 2024-03-09T14:09:06.770 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:09:07.771 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:09:07.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:09:08.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247810 -lt 98784247810 2024-03-09T14:09:08.049 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:09:08.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345923 2024-03-09T14:09:08.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:09:08.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:09:08.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345923 2024-03-09T14:09:08.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:09:08.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345923 2024-03-09T14:09:08.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345923' 2024-03-09T14:09:08.053 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 85899345923 2024-03-09T14:09:08.053 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:09:08.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345923 -lt 85899345923 2024-03-09T14:09:08.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:09:08.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:09:08.328 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:09:08.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T14:09:08.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:09:08.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:09:08.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:09:08.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:09:08.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:09:08.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:09:08.671 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:09:08.940 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T14:09:08.940 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:09:08.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:09:08.941 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:09:09.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T14:09:09.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:09:09.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:09:09.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_repair_stats: repair 1.0 2024-03-09T14:09:09.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-09T14:09:09.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-09T14:09:09.277 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T14:09:09.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:09:09.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:09:09.278 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T14:09:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T14:08:20.223387+0000 2024-03-09T14:09:09.547 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-09T14:09:09.807 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 1.0 on osd.1 to repair 2024-03-09T14:09:09.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-09T14:08:20.223387+0000 2024-03-09T14:09:09.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T14:09:09.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T14:08:20.223387+0000 2024-03-09T14:09:09.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T14:09:09.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T14:09:09.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:09:09.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T14:09:09.820 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T14:09:09.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:09:09.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:09:09.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T14:09:10.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:08:20.223387+0000 '>' 2024-03-09T14:08:20.223387+0000 2024-03-09T14:09:10.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:09:11.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:09:11.091 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:09:11.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T14:09:11.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T14:09:11.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:09:11.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:09:11.092 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T14:09:11.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:08:20.223387+0000 '>' 2024-03-09T14:08:20.223387+0000 2024-03-09T14:09:11.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:09:12.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:09:12.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:09:12.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T14:09:12.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T14:09:12.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:09:12.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:09:12.366 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T14:09:12.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:09:10.094325+0000 '>' 2024-03-09T14:08:20.223387+0000 2024-03-09T14:09:12.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T14:09:12.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_repair_stats: wait_for_clean 2024-03-09T14:09:12.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:09:12.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:09:12.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:09:12.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:09:12.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:09:12.643 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:09:12.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:09:12.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:09:12.643 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:09:12.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:09:12.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:09:12.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:09:12.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:09:12.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:09:12.735 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:09:13.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:09:13.006 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T14:09:13.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:09:13.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:09:13.006 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:09:13.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247812 2024-03-09T14:09:13.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247812 2024-03-09T14:09:13.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247812' 2024-03-09T14:09:13.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:09:13.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:09:13.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345925 2024-03-09T14:09:13.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345925 2024-03-09T14:09:13.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247812 1-85899345925' 2024-03-09T14:09:13.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:09:13.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247812 2024-03-09T14:09:13.215 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:09:13.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:09:13.216 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247812 2024-03-09T14:09:13.216 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:09:13.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247812 2024-03-09T14:09:13.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247812' 2024-03-09T14:09:13.218 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 98784247812 2024-03-09T14:09:13.218 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:09:13.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247811 -lt 98784247812 2024-03-09T14:09:13.493 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:09:14.494 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:09:14.494 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:09:14.774 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247812 -lt 98784247812 2024-03-09T14:09:14.775 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:09:14.775 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345925 2024-03-09T14:09:14.775 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:09:14.776 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:09:14.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345925 2024-03-09T14:09:14.777 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:09:14.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345925 2024-03-09T14:09:14.778 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345925' 2024-03-09T14:09:14.778 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 85899345925 2024-03-09T14:09:14.779 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:09:15.099 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345925 -lt 85899345925 2024-03-09T14:09:15.099 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:09:15.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:09:15.100 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:09:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T14:09:15.441 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:09:15.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:09:15.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:09:15.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:09:15.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:09:15.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:09:15.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:09:15.711 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T14:09:15.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:09:15.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:09:15.712 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:09:16.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T14:09:16.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:09:16.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:09:16.068 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:702: TEST_repair_stats: ceph pg dump pgs 2024-03-09T14:09:16.326 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T14:09:16.326 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-03-09T14:09:11.065765+0000 16'30 24:118 [1,0] 1 [1,0] 1 16'30 2024-03-09T14:09:10.094325+0000 16'30 2024-03-09T14:09:10.094325+0000 0 1 periodic scrub scheduled @ 2024-03-10T14:09:10.094325+0000 20 0 2024-03-09T14:09:16.326 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T14:09:16.327 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T14:09:16.327 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T14:09:16.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:703: TEST_repair_stats: flush_pg_stats 2024-03-09T14:09:16.337 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:09:16.338 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:09:16.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:09:16.609 INFO:tasks.workunit.client.0.smithi162.stderr:1' 2024-03-09T14:09:16.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:09:16.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:09:16.609 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:09:16.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=98784247814 2024-03-09T14:09:16.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 98784247814 2024-03-09T14:09:16.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247814' 2024-03-09T14:09:16.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:09:16.716 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:09:16.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345927 2024-03-09T14:09:16.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345927 2024-03-09T14:09:16.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-98784247814 1-85899345927' 2024-03-09T14:09:16.821 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:09:16.821 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-98784247814 2024-03-09T14:09:16.822 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:09:16.822 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:09:16.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-98784247814 2024-03-09T14:09:16.823 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:09:16.824 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=98784247814 2024-03-09T14:09:16.824 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 98784247814 2024-03-09T14:09:16.825 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 98784247814' 2024-03-09T14:09:16.825 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:09:17.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247813 -lt 98784247814 2024-03-09T14:09:17.096 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:09:18.097 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:09:18.098 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:09:18.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 98784247814 -lt 98784247814 2024-03-09T14:09:18.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:09:18.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-85899345927 2024-03-09T14:09:18.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:09:18.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:09:18.383 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-85899345927 2024-03-09T14:09:18.383 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:09:18.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345927 2024-03-09T14:09:18.385 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 85899345927 2024-03-09T14:09:18.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 85899345927' 2024-03-09T14:09:18.385 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:09:18.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345927 -lt 85899345927 2024-03-09T14:09:18.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: ceph pg 1.0 query 2024-03-09T14:09:18.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_repair_stats: jq .info.stats.stat_sum 2024-03-09T14:09:18.767 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T14:09:18.767 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 210, 2024-03-09T14:09:18.767 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 30, 2024-03-09T14:09:18.767 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-03-09T14:09:18.767 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 60, 2024-03-09T14:09:18.767 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-03-09T14:09:18.767 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 0, 2024-03-09T14:09:18.767 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 0, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 30, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 30, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 30, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 20, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 140, 2024-03-09T14:09:18.768 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-03-09T14:09:18.769 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-03-09T14:09:18.769 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-03-09T14:09:18.769 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-03-09T14:09:18.769 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-03-09T14:09:18.769 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-03-09T14:09:18.769 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-03-09T14:09:18.769 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-03-09T14:09:18.769 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-03-09T14:09:18.769 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-03-09T14:09:18.769 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-03-09T14:09:18.769 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-03-09T14:09:18.769 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-03-09T14:09:18.770 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-03-09T14:09:18.770 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-03-09T14:09:18.770 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-03-09T14:09:18.770 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-03-09T14:09:18.770 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-03-09T14:09:18.770 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-03-09T14:09:18.770 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 20 2024-03-09T14:09:18.770 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T14:09:18.770 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: ceph pg 1.0 query 2024-03-09T14:09:18.771 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2024-03-09T14:09:18.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_repair_stats: COUNT=20 2024-03-09T14:09:18.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_repair_stats: test 20 = 20 2024-03-09T14:09:18.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-09T14:09:18.875 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:710: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2024-03-09T14:09:19.132 INFO:tasks.workunit.client.0.smithi162.stderr:dumped all 2024-03-09T14:09:19.143 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T14:09:19.143 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": 1, 2024-03-09T14:09:19.143 INFO:tasks.workunit.client.0.smithi162.stdout: "up_from": 20, 2024-03-09T14:09:19.143 INFO:tasks.workunit.client.0.smithi162.stdout: "seq": 85899345927, 2024-03-09T14:09:19.143 INFO:tasks.workunit.client.0.smithi162.stdout: "num_pgs": 1, 2024-03-09T14:09:19.143 INFO:tasks.workunit.client.0.smithi162.stdout: "num_osds": 1, 2024-03-09T14:09:19.144 INFO:tasks.workunit.client.0.smithi162.stdout: "num_per_pool_osds": 1, 2024-03-09T14:09:19.144 INFO:tasks.workunit.client.0.smithi162.stdout: "num_per_pool_omap_osds": 1, 2024-03-09T14:09:19.144 INFO:tasks.workunit.client.0.smithi162.stdout: "kb": 104857600, 2024-03-09T14:09:19.144 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used": 28552, 2024-03-09T14:09:19.144 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used_data": 320, 2024-03-09T14:09:19.144 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used_omap": 4, 2024-03-09T14:09:19.144 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used_meta": 28219, 2024-03-09T14:09:19.144 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_avail": 104829048, 2024-03-09T14:09:19.144 INFO:tasks.workunit.client.0.smithi162.stdout: "statfs": { 2024-03-09T14:09:19.144 INFO:tasks.workunit.client.0.smithi162.stdout: "total": 107374182400, 2024-03-09T14:09:19.145 INFO:tasks.workunit.client.0.smithi162.stdout: "available": 107344945152, 2024-03-09T14:09:19.145 INFO:tasks.workunit.client.0.smithi162.stdout: "internally_reserved": 0, 2024-03-09T14:09:19.145 INFO:tasks.workunit.client.0.smithi162.stdout: "allocated": 327680, 2024-03-09T14:09:19.145 INFO:tasks.workunit.client.0.smithi162.stdout: "data_stored": 53726, 2024-03-09T14:09:19.145 INFO:tasks.workunit.client.0.smithi162.stdout: "data_compressed": 0, 2024-03-09T14:09:19.145 INFO:tasks.workunit.client.0.smithi162.stdout: "data_compressed_allocated": 0, 2024-03-09T14:09:19.145 INFO:tasks.workunit.client.0.smithi162.stdout: "data_compressed_original": 0, 2024-03-09T14:09:19.145 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_allocated": 4398, 2024-03-09T14:09:19.145 INFO:tasks.workunit.client.0.smithi162.stdout: "internal_metadata": 28896978 2024-03-09T14:09:19.145 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T14:09:19.145 INFO:tasks.workunit.client.0.smithi162.stdout: "hb_peers": [ 2024-03-09T14:09:19.145 INFO:tasks.workunit.client.0.smithi162.stdout: 0 2024-03-09T14:09:19.145 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-03-09T14:09:19.146 INFO:tasks.workunit.client.0.smithi162.stdout: "snap_trim_queue_len": 0, 2024-03-09T14:09:19.146 INFO:tasks.workunit.client.0.smithi162.stdout: "num_snap_trimming": 0, 2024-03-09T14:09:19.146 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shards_repaired": 10, 2024-03-09T14:09:19.146 INFO:tasks.workunit.client.0.smithi162.stdout: "op_queue_age_hist": { 2024-03-09T14:09:19.146 INFO:tasks.workunit.client.0.smithi162.stdout: "histogram": [], 2024-03-09T14:09:19.146 INFO:tasks.workunit.client.0.smithi162.stdout: "upper_bound": 1 2024-03-09T14:09:19.146 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T14:09:19.146 INFO:tasks.workunit.client.0.smithi162.stdout: "perf_stat": { 2024-03-09T14:09:19.146 INFO:tasks.workunit.client.0.smithi162.stdout: "commit_latency_ms": 0, 2024-03-09T14:09:19.146 INFO:tasks.workunit.client.0.smithi162.stdout: "apply_latency_ms": 0, 2024-03-09T14:09:19.147 INFO:tasks.workunit.client.0.smithi162.stdout: "commit_latency_ns": 0, 2024-03-09T14:09:19.147 INFO:tasks.workunit.client.0.smithi162.stdout: "apply_latency_ns": 0 2024-03-09T14:09:19.147 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T14:09:19.147 INFO:tasks.workunit.client.0.smithi162.stdout: "alerts": [] 2024-03-09T14:09:19.147 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T14:09:19.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-09T14:09:19.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2024-03-09T14:09:19.399 INFO:tasks.workunit.client.0.smithi162.stderr:dumped all 2024-03-09T14:09:19.410 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:711: TEST_repair_stats: COUNT=10 2024-03-09T14:09:19.411 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: expr 20 / 2 2024-03-09T14:09:19.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:712: TEST_repair_stats: test 10 = 10 2024-03-09T14:09:19.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-09T14:09:19.412 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-03-09T14:09:19.667 INFO:tasks.workunit.client.0.smithi162.stderr:dumped all 2024-03-09T14:09:19.679 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T14:09:19.679 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": 0, 2024-03-09T14:09:19.679 INFO:tasks.workunit.client.0.smithi162.stdout: "up_from": 23, 2024-03-09T14:09:19.679 INFO:tasks.workunit.client.0.smithi162.stdout: "seq": 98784247815, 2024-03-09T14:09:19.679 INFO:tasks.workunit.client.0.smithi162.stdout: "num_pgs": 1, 2024-03-09T14:09:19.679 INFO:tasks.workunit.client.0.smithi162.stdout: "num_osds": 1, 2024-03-09T14:09:19.679 INFO:tasks.workunit.client.0.smithi162.stdout: "num_per_pool_osds": 1, 2024-03-09T14:09:19.679 INFO:tasks.workunit.client.0.smithi162.stdout: "num_per_pool_omap_osds": 1, 2024-03-09T14:09:19.679 INFO:tasks.workunit.client.0.smithi162.stdout: "kb": 104857600, 2024-03-09T14:09:19.680 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used": 28552, 2024-03-09T14:09:19.680 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used_data": 320, 2024-03-09T14:09:19.680 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used_omap": 4, 2024-03-09T14:09:19.680 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used_meta": 28219, 2024-03-09T14:09:19.680 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_avail": 104829048, 2024-03-09T14:09:19.680 INFO:tasks.workunit.client.0.smithi162.stdout: "statfs": { 2024-03-09T14:09:19.680 INFO:tasks.workunit.client.0.smithi162.stdout: "total": 107374182400, 2024-03-09T14:09:19.680 INFO:tasks.workunit.client.0.smithi162.stdout: "available": 107344945152, 2024-03-09T14:09:19.680 INFO:tasks.workunit.client.0.smithi162.stdout: "internally_reserved": 0, 2024-03-09T14:09:19.680 INFO:tasks.workunit.client.0.smithi162.stdout: "allocated": 327680, 2024-03-09T14:09:19.680 INFO:tasks.workunit.client.0.smithi162.stdout: "data_stored": 53726, 2024-03-09T14:09:19.681 INFO:tasks.workunit.client.0.smithi162.stdout: "data_compressed": 0, 2024-03-09T14:09:19.681 INFO:tasks.workunit.client.0.smithi162.stdout: "data_compressed_allocated": 0, 2024-03-09T14:09:19.681 INFO:tasks.workunit.client.0.smithi162.stdout: "data_compressed_original": 0, 2024-03-09T14:09:19.681 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_allocated": 4401, 2024-03-09T14:09:19.681 INFO:tasks.workunit.client.0.smithi162.stdout: "internal_metadata": 28896975 2024-03-09T14:09:19.681 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T14:09:19.681 INFO:tasks.workunit.client.0.smithi162.stdout: "hb_peers": [ 2024-03-09T14:09:19.681 INFO:tasks.workunit.client.0.smithi162.stdout: 1 2024-03-09T14:09:19.681 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-03-09T14:09:19.682 INFO:tasks.workunit.client.0.smithi162.stdout: "snap_trim_queue_len": 0, 2024-03-09T14:09:19.682 INFO:tasks.workunit.client.0.smithi162.stdout: "num_snap_trimming": 0, 2024-03-09T14:09:19.682 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shards_repaired": 10, 2024-03-09T14:09:19.682 INFO:tasks.workunit.client.0.smithi162.stdout: "op_queue_age_hist": { 2024-03-09T14:09:19.682 INFO:tasks.workunit.client.0.smithi162.stdout: "histogram": [], 2024-03-09T14:09:19.682 INFO:tasks.workunit.client.0.smithi162.stdout: "upper_bound": 1 2024-03-09T14:09:19.682 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T14:09:19.682 INFO:tasks.workunit.client.0.smithi162.stdout: "perf_stat": { 2024-03-09T14:09:19.682 INFO:tasks.workunit.client.0.smithi162.stdout: "commit_latency_ms": 0, 2024-03-09T14:09:19.682 INFO:tasks.workunit.client.0.smithi162.stdout: "apply_latency_ms": 0, 2024-03-09T14:09:19.682 INFO:tasks.workunit.client.0.smithi162.stdout: "commit_latency_ns": 0, 2024-03-09T14:09:19.682 INFO:tasks.workunit.client.0.smithi162.stdout: "apply_latency_ns": 0 2024-03-09T14:09:19.683 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T14:09:19.683 INFO:tasks.workunit.client.0.smithi162.stdout: "alerts": [] 2024-03-09T14:09:19.683 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T14:09:19.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-09T14:09:19.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-03-09T14:09:19.973 INFO:tasks.workunit.client.0.smithi162.stderr:dumped all 2024-03-09T14:09:19.985 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: COUNT=10 2024-03-09T14:09:19.985 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: expr 20 / 2 2024-03-09T14:09:19.986 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: test 10 = 10 2024-03-09T14:09:19.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-09T14:09:19.987 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: jq .pg_map.osd_stats_sum 2024-03-09T14:09:20.244 INFO:tasks.workunit.client.0.smithi162.stderr:dumped all 2024-03-09T14:09:20.255 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T14:09:20.255 INFO:tasks.workunit.client.0.smithi162.stdout: "up_from": 0, 2024-03-09T14:09:20.255 INFO:tasks.workunit.client.0.smithi162.stdout: "seq": 0, 2024-03-09T14:09:20.255 INFO:tasks.workunit.client.0.smithi162.stdout: "num_pgs": 2, 2024-03-09T14:09:20.255 INFO:tasks.workunit.client.0.smithi162.stdout: "num_osds": 2, 2024-03-09T14:09:20.256 INFO:tasks.workunit.client.0.smithi162.stdout: "num_per_pool_osds": 2, 2024-03-09T14:09:20.256 INFO:tasks.workunit.client.0.smithi162.stdout: "num_per_pool_omap_osds": 2, 2024-03-09T14:09:20.256 INFO:tasks.workunit.client.0.smithi162.stdout: "kb": 209715200, 2024-03-09T14:09:20.256 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used": 57104, 2024-03-09T14:09:20.256 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used_data": 640, 2024-03-09T14:09:20.256 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used_omap": 8, 2024-03-09T14:09:20.256 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used_meta": 56439, 2024-03-09T14:09:20.256 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_avail": 209658096, 2024-03-09T14:09:20.256 INFO:tasks.workunit.client.0.smithi162.stdout: "statfs": { 2024-03-09T14:09:20.256 INFO:tasks.workunit.client.0.smithi162.stdout: "total": 214748364800, 2024-03-09T14:09:20.256 INFO:tasks.workunit.client.0.smithi162.stdout: "available": 214689890304, 2024-03-09T14:09:20.257 INFO:tasks.workunit.client.0.smithi162.stdout: "internally_reserved": 0, 2024-03-09T14:09:20.257 INFO:tasks.workunit.client.0.smithi162.stdout: "allocated": 655360, 2024-03-09T14:09:20.257 INFO:tasks.workunit.client.0.smithi162.stdout: "data_stored": 107452, 2024-03-09T14:09:20.257 INFO:tasks.workunit.client.0.smithi162.stdout: "data_compressed": 0, 2024-03-09T14:09:20.257 INFO:tasks.workunit.client.0.smithi162.stdout: "data_compressed_allocated": 0, 2024-03-09T14:09:20.257 INFO:tasks.workunit.client.0.smithi162.stdout: "data_compressed_original": 0, 2024-03-09T14:09:20.257 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_allocated": 8799, 2024-03-09T14:09:20.257 INFO:tasks.workunit.client.0.smithi162.stdout: "internal_metadata": 57793953 2024-03-09T14:09:20.257 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T14:09:20.257 INFO:tasks.workunit.client.0.smithi162.stdout: "hb_peers": [], 2024-03-09T14:09:20.257 INFO:tasks.workunit.client.0.smithi162.stdout: "snap_trim_queue_len": 0, 2024-03-09T14:09:20.257 INFO:tasks.workunit.client.0.smithi162.stdout: "num_snap_trimming": 0, 2024-03-09T14:09:20.258 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shards_repaired": 20, 2024-03-09T14:09:20.258 INFO:tasks.workunit.client.0.smithi162.stdout: "op_queue_age_hist": { 2024-03-09T14:09:20.258 INFO:tasks.workunit.client.0.smithi162.stdout: "histogram": [], 2024-03-09T14:09:20.258 INFO:tasks.workunit.client.0.smithi162.stdout: "upper_bound": 1 2024-03-09T14:09:20.258 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T14:09:20.258 INFO:tasks.workunit.client.0.smithi162.stdout: "perf_stat": { 2024-03-09T14:09:20.258 INFO:tasks.workunit.client.0.smithi162.stdout: "commit_latency_ms": 0, 2024-03-09T14:09:20.258 INFO:tasks.workunit.client.0.smithi162.stdout: "apply_latency_ms": 0, 2024-03-09T14:09:20.258 INFO:tasks.workunit.client.0.smithi162.stdout: "commit_latency_ns": 0, 2024-03-09T14:09:20.258 INFO:tasks.workunit.client.0.smithi162.stdout: "apply_latency_ns": 0 2024-03-09T14:09:20.258 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T14:09:20.259 INFO:tasks.workunit.client.0.smithi162.stdout: "alerts": [], 2024-03-09T14:09:20.259 INFO:tasks.workunit.client.0.smithi162.stdout: "network_ping_times": [] 2024-03-09T14:09:20.259 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T14:09:20.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: ceph pg dump --format=json-pretty 2024-03-09T14:09:20.259 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-03-09T14:09:20.514 INFO:tasks.workunit.client.0.smithi162.stderr:dumped all 2024-03-09T14:09:20.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:719: TEST_repair_stats: COUNT=20 2024-03-09T14:09:20.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:720: TEST_repair_stats: test 20 = 20 2024-03-09T14:09:20.530 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2024-03-09T14:09:20.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T14:09:20.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T14:09:20.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T14:09:20.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:09:20.531 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:09:20.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:09:20.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:09:20.531 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:09:20.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:09:20.635 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T14:09:20.636 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T14:09:20.636 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T14:09:20.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T14:09:20.637 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T14:09:20.638 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T14:09:20.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:09:20.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T14:09:20.639 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T14:09:20.639 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:09:20.640 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T14:09:20.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T14:09:20.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T14:09:20.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T14:09:20.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:09:20.654 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:09:20.654 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T14:09:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T14:09:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T14:09:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2024-03-09T14:09:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2024-03-09T14:09:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-scrub-repair 2024-03-09T14:09:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-scrub-repair 2024-03-09T14:09:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T14:09:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-03-09T14:09:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T14:09:20.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:09:20.656 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:09:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:09:20.656 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:09:20.657 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:09:20.658 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:09:20.658 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T14:09:20.659 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T14:09:20.659 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T14:09:20.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T14:09:20.660 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T14:09:20.661 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T14:09:20.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:09:20.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T14:09:20.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T14:09:20.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:09:20.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T14:09:20.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-03-09T14:09:20.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T14:09:20.665 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T14:09:20.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:09:20.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:09:20.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T14:09:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T14:09:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T14:09:20.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-scrub-repair 2024-03-09T14:09:20.668 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-03-09T14:09:20.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:09:20.669 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:09:20.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.70042 2024-03-09T14:09:20.673 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-03-09T14:09:20.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-03-09T14:09:20.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-03-09T14:09:20.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2024-03-09T14:09:20.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2024-03-09T14:09:20.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2024-03-09T14:09:20.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats_ec: local poolname=testpool 2024-03-09T14:09:20.673 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats_ec: local OSDS=3 2024-03-09T14:09:20.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:727: TEST_repair_stats_ec: local OBJS=30 2024-03-09T14:09:20.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats_ec: local REPAIRS=26 2024-03-09T14:09:20.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats_ec: local allow_overwrites=false 2024-03-09T14:09:20.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:733: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2024-03-09T14:09:20.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-scrub-repair 2024-03-09T14:09:20.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-03-09T14:09:20.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-03-09T14:09:20.674 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-03-09T14:09:20.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-scrub-repair/a 2024-03-09T14:09:20.675 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2024-03-09T14:09:21.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-03-09T14:09:21.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:09:21.143 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:09:21.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:09:21.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:09:21.143 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:09:21.145 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:09:21.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-03-09T14:09:21.348 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-03-09T14:09:21.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-03-09T14:09:21.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T14:09:21.349 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T14:09:21.350 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-03-09T14:09:21.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-03-09T14:09:21.351 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T14:09:21.351 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T14:09:21.351 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T14:09:21.351 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T14:09:21.351 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:09:21.351 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:09:21.351 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T14:09:21.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T14:09:21.351 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get fsid 2024-03-09T14:09:21.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-03-09T14:09:21.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-03-09T14:09:21.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-03-09T14:09:21.432 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-03-09T14:09:21.433 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-03-09T14:09:21.433 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-03-09T14:09:21.434 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-03-09T14:09:21.434 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-03-09T14:09:21.434 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-03-09T14:09:21.434 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:09:21.434 INFO:tasks.workunit.client.0.smithi162.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:09:21.434 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.70042/ceph-mon.a.asok 2024-03-09T14:09:21.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-03-09T14:09:21.434 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.70042/ceph-mon.a.asok config get mon_host 2024-03-09T14:09:21.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2024-03-09T14:09:21.513 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-scrub-repair 2024-03-09T14:09:21.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-03-09T14:09:21.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-03-09T14:09:21.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-03-09T14:09:21.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-scrub-repair/x 2024-03-09T14:09:21.514 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-03-09T14:09:22.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-03-09T14:09:22.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:09:22.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:09:22.326 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:09:22.326 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:09:22.326 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:09:22.326 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:09:22.327 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T14:09:22.327 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-03-09T14:09:22.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T14:09:22.355 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: expr 3 - 1 2024-03-09T14:09:22.356 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: seq 0 2 2024-03-09T14:09:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T14:09:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:09:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T14:09:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T14:09:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-03-09T14:09:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T14:09:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:09:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:09:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:09:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:09:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:09:22.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:09:22.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T14:09:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T14:09:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:09:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:09:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:09:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:09:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:09:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:09:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T14:09:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T14:09:22.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T14:09:22.363 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T14:09:22.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=66d1bcfe-667e-4ef5-a21b-5224e96e1898 2024-03-09T14:09:22.364 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 66d1bcfe-667e-4ef5-a21b-5224e96e1898' 2024-03-09T14:09:22.364 INFO:tasks.workunit.client.0.smithi162.stdout:add osd0 66d1bcfe-667e-4ef5-a21b-5224e96e1898 2024-03-09T14:09:22.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T14:09:22.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCSbexlZouhFhAAhX9LECIk7H8fzggu2NAAZQ== 2024-03-09T14:09:22.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCSbexlZouhFhAAhX9LECIk7H8fzggu2NAAZQ=="}' 2024-03-09T14:09:22.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 66d1bcfe-667e-4ef5-a21b-5224e96e1898 -i td/osd-scrub-repair/0/new.json 2024-03-09T14:09:22.581 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:09:22.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/0/new.json 2024-03-09T14:09:22.592 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCSbexlZouhFhAAhX9LECIk7H8fzggu2NAAZQ== --osd-uuid 66d1bcfe-667e-4ef5-a21b-5224e96e1898 2024-03-09T14:09:22.621 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:22.620+0000 7fe17916b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:09:22.632 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:22.632+0000 7fe17916b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:09:22.633 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:22.633+0000 7fe17916b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:09:22.634 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:22.633+0000 7fe17916b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:09:22.634 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:22.633+0000 7fe17916b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:09:22.634 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:22.633+0000 7fe17916b640 -1 bluestore(td/osd-scrub-repair/0/block) _read_bdev_label failed to open td/osd-scrub-repair/0/block: (2) No such file or directory 2024-03-09T14:09:22.724 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:22.723+0000 7fe17916b640 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2024-03-09T14:09:28.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2024-03-09T14:09:28.449 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T14:09:28.450 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd0 key to auth repository 2024-03-09T14:09:28.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-03-09T14:09:28.450 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T14:09:28.804 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T14:09:28.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-03-09T14:09:28.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:09:28.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T14:09:28.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T14:09:28.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T14:09:28.835 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:28.831+0000 7f9c77211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:09:28.835 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:28.833+0000 7f9c77211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:09:28.835 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:28.834+0000 7f9c77211640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:09:29.083 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-03-09T14:09:29.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:09:29.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T14:09:29.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:09:29.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:09:29.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:29.084 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:09:29.084 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:09:29.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:29.085 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:29.353 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:29.352+0000 7f9c77211640 -1 Falling back to public interface 2024-03-09T14:09:29.359 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:30.360 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:30.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:30.361 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:09:30.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:09:30.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:30.361 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:30.641 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:31.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:31.642 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:31.642 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:09:31.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:09:31.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:31.643 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:31.711 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:31.710+0000 7f9c77211640 -1 osd.0 0 log_to_monitors true 2024-03-09T14:09:31.977 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:32.978 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:32.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:32.979 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:09:32.979 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:09:32.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:32.980 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:33.290 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:34.291 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:34.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:34.292 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T14:09:34.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T14:09:34.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:34.292 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:34.649 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:35.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:35.650 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:35.650 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T14:09:35.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T14:09:35.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:35.651 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:35.934 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:36.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:36.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:36.936 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T14:09:36.936 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T14:09:36.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:36.937 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:37.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:38.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:38.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:38.281 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T14:09:38.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T14:09:38.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:38.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:38.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:39.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:39.662 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:39.662 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T14:09:39.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T14:09:39.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:39.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:39.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:40.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:40.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:40.946 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T14:09:40.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T14:09:40.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:40.947 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:41.276 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:42.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:42.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:42.278 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T14:09:42.278 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T14:09:42.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:42.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:42.561 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:43.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:43.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:43.563 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T14:09:43.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T14:09:43.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:43.564 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:43.943 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:44.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:44.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:44.945 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T14:09:44.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T14:09:44.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:44.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:45.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:46.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:46.538 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:46.539 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T14:09:46.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T14:09:46.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:46.539 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:47.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:48.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:48.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:48.383 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T14:09:48.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T14:09:48.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:48.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:48.891 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:49.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:49.893 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:49.893 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T14:09:49.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T14:09:49.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:49.894 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:50.669 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:51.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:51.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:51.670 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T14:09:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T14:09:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:51.671 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:52.154 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:53.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:53.156 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:53.156 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T14:09:53.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T14:09:53.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:53.157 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:53.856 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:09:54.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:09:54.858 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:09:54.858 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T14:09:54.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T14:09:54.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:09:54.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:09:55.142 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/895875688,v1:127.0.0.1:6803/895875688] [v2:127.0.0.1:6804/895875688,v1:127.0.0.1:6805/895875688] exists,up 66d1bcfe-667e-4ef5-a21b-5224e96e1898 2024-03-09T14:09:55.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:09:55.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:09:55.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:09:55.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T14:09:55.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:09:55.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T14:09:55.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T14:09:55.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-03-09T14:09:55.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T14:09:55.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:09:55.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:09:55.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T14:09:55.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T14:09:55.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e6322f9f-8376-4351-962b-cb56b8af7f58 2024-03-09T14:09:55.149 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e6322f9f-8376-4351-962b-cb56b8af7f58' 2024-03-09T14:09:55.149 INFO:tasks.workunit.client.0.smithi162.stdout:add osd1 e6322f9f-8376-4351-962b-cb56b8af7f58 2024-03-09T14:09:55.150 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T14:09:55.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCzbexlQ4jGCRAAA/Fw2WqFpDWQww9MSVEvHQ== 2024-03-09T14:09:55.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCzbexlQ4jGCRAAA/Fw2WqFpDWQww9MSVEvHQ=="}' 2024-03-09T14:09:55.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e6322f9f-8376-4351-962b-cb56b8af7f58 -i td/osd-scrub-repair/1/new.json 2024-03-09T14:09:55.493 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:09:55.504 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/1/new.json 2024-03-09T14:09:55.505 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCzbexlQ4jGCRAAA/Fw2WqFpDWQww9MSVEvHQ== --osd-uuid e6322f9f-8376-4351-962b-cb56b8af7f58 2024-03-09T14:09:55.533 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:55.534+0000 7f4a4b56d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:09:55.581 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:55.577+0000 7f4a4b56d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:09:55.582 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:55.582+0000 7f4a4b56d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:09:55.582 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:55.583+0000 7f4a4b56d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:09:55.582 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:55.583+0000 7f4a4b56d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:09:55.582 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:55.583+0000 7f4a4b56d640 -1 bluestore(td/osd-scrub-repair/1/block) _read_bdev_label failed to open td/osd-scrub-repair/1/block: (2) No such file or directory 2024-03-09T14:09:55.685 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:09:55.686+0000 7f4a4b56d640 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2024-03-09T14:10:01.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2024-03-09T14:10:01.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T14:10:01.806 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd1 key to auth repository 2024-03-09T14:10:01.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-03-09T14:10:01.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T14:10:02.163 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T14:10:02.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-03-09T14:10:02.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:10:02.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T14:10:02.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T14:10:02.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T14:10:02.191 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:02.190+0000 7f3389e17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:10:02.193 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:02.192+0000 7f3389e17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:10:02.194 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:02.193+0000 7f3389e17640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:10:02.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-03-09T14:10:02.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:10:02.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T14:10:02.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:10:02.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:10:02.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:02.470 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:10:02.471 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:10:02.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:02.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:02.720 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:02.719+0000 7f3389e17640 -1 Falling back to public interface 2024-03-09T14:10:02.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:03.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:03.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:03.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:10:03.747 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:10:03.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:03.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:04.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:05.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:05.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:05.031 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:10:05.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:10:05.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:05.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:05.123 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:05.122+0000 7f3389e17640 -1 osd.1 0 log_to_monitors true 2024-03-09T14:10:05.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:06.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:06.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:06.331 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:10:06.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:10:06.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:06.332 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:06.745 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:07.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:07.746 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:07.747 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T14:10:07.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T14:10:07.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:07.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:08.029 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:09.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:09.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:09.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T14:10:09.031 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T14:10:09.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:09.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:09.365 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:10.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:10.367 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:10.367 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T14:10:10.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T14:10:10.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:10.368 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:10.754 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:11.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:11.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:11.756 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T14:10:11.756 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T14:10:11.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:11.757 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:12.042 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:13.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:13.043 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:13.044 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T14:10:13.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T14:10:13.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:13.044 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:13.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:14.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:14.320 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:14.320 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T14:10:14.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T14:10:14.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:14.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:14.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:15.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:15.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:15.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T14:10:15.734 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T14:10:15.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:15.735 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:16.016 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:17.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:17.018 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:17.018 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T14:10:17.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T14:10:17.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:17.019 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:17.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:18.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:18.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:18.307 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T14:10:18.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T14:10:18.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:18.308 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:19.030 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:20.031 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:20.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:20.032 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T14:10:20.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T14:10:20.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:20.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:20.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:21.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:21.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:21.317 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T14:10:21.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T14:10:21.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:21.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:21.849 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:22.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:22.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:22.851 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T14:10:22.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T14:10:22.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:22.852 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:23.594 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:24.595 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:24.596 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T14:10:24.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:24.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T14:10:24.596 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:24.597 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:25.404 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:26.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:26.406 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:26.406 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T14:10:26.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T14:10:26.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:26.407 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:27.215 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:28.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:28.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:28.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T14:10:28.217 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T14:10:28.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:28.218 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:28.541 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:29.543 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:29.543 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T14:10:29.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:29.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T14:10:29.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:29.544 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:10:29.827 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/837939467,v1:127.0.0.1:6811/837939467] [v2:127.0.0.1:6812/837939467,v1:127.0.0.1:6813/837939467] exists,up e6322f9f-8376-4351-962b-cb56b8af7f58 2024-03-09T14:10:29.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:10:29.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:10:29.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:10:29.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:737: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T14:10:29.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:10:29.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-scrub-repair 2024-03-09T14:10:29.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-03-09T14:10:29.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-scrub-repair/2 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:10:29.833 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:10:29.834 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:10:29.834 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:10:29.834 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:10:29.834 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:10:29.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:10:29.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:10:29.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-03-09T14:10:29.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-03-09T14:10:29.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:10:29.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:10:29.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:10:29.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:10:29.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:10:29.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:10:29.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-03-09T14:10:29.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T14:10:29.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-scrub-repair/2 2024-03-09T14:10:29.835 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-03-09T14:10:29.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1d60d0fb-595a-4deb-a947-b77e9d4ef2ce 2024-03-09T14:10:29.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1d60d0fb-595a-4deb-a947-b77e9d4ef2ce' 2024-03-09T14:10:29.835 INFO:tasks.workunit.client.0.smithi162.stdout:add osd2 1d60d0fb-595a-4deb-a947-b77e9d4ef2ce 2024-03-09T14:10:29.836 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-03-09T14:10:29.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVbexludG3MhAAKngcEo/D7fcX2rYMKy7vag== 2024-03-09T14:10:29.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVbexludG3MhAAKngcEo/D7fcX2rYMKy7vag=="}' 2024-03-09T14:10:29.851 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1d60d0fb-595a-4deb-a947-b77e9d4ef2ce -i td/osd-scrub-repair/2/new.json 2024-03-09T14:10:30.236 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:10:30.247 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-scrub-repair/2/new.json 2024-03-09T14:10:30.248 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDVbexludG3MhAAKngcEo/D7fcX2rYMKy7vag== --osd-uuid 1d60d0fb-595a-4deb-a947-b77e9d4ef2ce 2024-03-09T14:10:30.277 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:30.278+0000 7f6be9424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:10:30.321 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:30.321+0000 7f6be9424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:10:30.322 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:30.322+0000 7f6be9424640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:10:30.322 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:30.322+0000 7f6be9424640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T14:10:30.322 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:30.323+0000 7f6be9424640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T14:10:30.322 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:30.323+0000 7f6be9424640 -1 bluestore(td/osd-scrub-repair/2/block) _read_bdev_label failed to open td/osd-scrub-repair/2/block: (2) No such file or directory 2024-03-09T14:10:30.387 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:30.387+0000 7f6be9424640 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2024-03-09T14:10:36.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2024-03-09T14:10:36.376 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-03-09T14:10:36.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-03-09T14:10:36.377 INFO:tasks.workunit.client.0.smithi162.stdout:adding osd2 key to auth repository 2024-03-09T14:10:36.377 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-03-09T14:10:36.731 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.2 2024-03-09T14:10:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-03-09T14:10:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:10:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-03-09T14:10:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-03-09T14:10:36.732 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-03-09T14:10:36.758 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:36.759+0000 7ff3a3ec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:10:36.760 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:36.761+0000 7ff3a3ec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:10:36.762 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:36.762+0000 7ff3a3ec6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:10:37.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-03-09T14:10:37.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:10:37.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-03-09T14:10:37.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:10:37.011 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:10:37.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:37.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:10:37.012 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:10:37.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:37.012 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:37.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:37.294 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:37.295+0000 7ff3a3ec6640 -1 Falling back to public interface 2024-03-09T14:10:38.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:38.285 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:38.285 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:10:38.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:10:38.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:38.286 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:38.560 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:39.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:39.562 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:39.562 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:10:39.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:10:39.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:39.563 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:39.685 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:10:39.685+0000 7ff3a3ec6640 -1 osd.2 0 log_to_monitors true 2024-03-09T14:10:39.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:40.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:40.895 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:40.895 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:10:40.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:10:40.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:40.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:41.178 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:42.179 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:42.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:42.180 INFO:tasks.workunit.client.0.smithi162.stdout:4 2024-03-09T14:10:42.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-03-09T14:10:42.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:42.180 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:42.586 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:43.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:43.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:43.588 INFO:tasks.workunit.client.0.smithi162.stdout:5 2024-03-09T14:10:43.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-03-09T14:10:43.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:43.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:43.921 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:44.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:44.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:44.922 INFO:tasks.workunit.client.0.smithi162.stdout:6 2024-03-09T14:10:44.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-03-09T14:10:44.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:44.923 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:45.207 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:46.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:46.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:46.208 INFO:tasks.workunit.client.0.smithi162.stdout:7 2024-03-09T14:10:46.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-03-09T14:10:46.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:46.209 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:46.520 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:47.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:47.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:47.522 INFO:tasks.workunit.client.0.smithi162.stdout:8 2024-03-09T14:10:47.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-03-09T14:10:47.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:47.523 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:47.803 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:48.805 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:48.886 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:48.886 INFO:tasks.workunit.client.0.smithi162.stdout:9 2024-03-09T14:10:48.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-03-09T14:10:48.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:48.887 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:49.090 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:50.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:50.092 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:50.092 INFO:tasks.workunit.client.0.smithi162.stdout:10 2024-03-09T14:10:50.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-03-09T14:10:50.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:50.093 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:50.454 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:51.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:51.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:51.456 INFO:tasks.workunit.client.0.smithi162.stdout:11 2024-03-09T14:10:51.456 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-03-09T14:10:51.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:51.457 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:51.738 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:52.739 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:52.739 INFO:tasks.workunit.client.0.smithi162.stdout:12 2024-03-09T14:10:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-03-09T14:10:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:52.740 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:53.328 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:54.329 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:54.330 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:54.330 INFO:tasks.workunit.client.0.smithi162.stdout:13 2024-03-09T14:10:54.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-03-09T14:10:54.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:54.331 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:55.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:56.281 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:56.282 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:56.282 INFO:tasks.workunit.client.0.smithi162.stdout:14 2024-03-09T14:10:56.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-03-09T14:10:56.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:56.283 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:56.624 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:57.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:57.625 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:57.625 INFO:tasks.workunit.client.0.smithi162.stdout:15 2024-03-09T14:10:57.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-03-09T14:10:57.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:57.626 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:58.102 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:10:59.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:10:59.104 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:10:59.104 INFO:tasks.workunit.client.0.smithi162.stdout:16 2024-03-09T14:10:59.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-03-09T14:10:59.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:10:59.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:10:59.480 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:11:00.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:11:00.481 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:11:00.482 INFO:tasks.workunit.client.0.smithi162.stdout:17 2024-03-09T14:11:00.482 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-03-09T14:11:00.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:11:00.483 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:11:01.316 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:11:02.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:11:02.317 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:11:02.318 INFO:tasks.workunit.client.0.smithi162.stdout:18 2024-03-09T14:11:02.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-03-09T14:11:02.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:11:02.318 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:11:02.818 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:11:03.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:11:03.819 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:11:03.820 INFO:tasks.workunit.client.0.smithi162.stdout:19 2024-03-09T14:11:03.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-03-09T14:11:03.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:11:03.820 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-03-09T14:11:04.118 INFO:tasks.workunit.client.0.smithi162.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2875006573,v1:127.0.0.1:6819/2875006573] [v2:127.0.0.1:6820/2875006573,v1:127.0.0.1:6821/2875006573] exists,up 1d60d0fb-595a-4deb-a947-b77e9d4ef2ce 2024-03-09T14:11:04.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:11:04.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:11:04.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:11:04.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:742: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2024-03-09T14:11:04.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2435: create_ec_pool: local pool_name=testpool 2024-03-09T14:11:04.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2436: create_ec_pool: shift 2024-03-09T14:11:04.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2437: create_ec_pool: local allow_overwrites=false 2024-03-09T14:11:04.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2438: create_ec_pool: shift 2024-03-09T14:11:04.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2440: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-03-09T14:11:04.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2442: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2024-03-09T14:11:04.527 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2024-03-09T14:11:05.119 INFO:tasks.workunit.client.0.smithi162.stderr:pool 'testpool' created 2024-03-09T14:11:05.130 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-03-09T14:11:06.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: '[' false = true ']' 2024-03-09T14:11:06.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: wait_for_clean 2024-03-09T14:11:06.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:11:06.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:11:06.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:11:06.133 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:11:06.133 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:11:06.134 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:11:06.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:11:06.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:11:06.134 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:11:06.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:11:06.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:11:06.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:11:06.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:11:06.237 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:11:06.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:11:06.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:11:06.577 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:11:06.577 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T14:11:06.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:11:06.577 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:11:06.577 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:11:06.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836502 2024-03-09T14:11:06.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836502 2024-03-09T14:11:06.680 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502' 2024-03-09T14:11:06.681 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:11:06.681 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:11:06.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672969 2024-03-09T14:11:06.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672969 2024-03-09T14:11:06.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672969' 2024-03-09T14:11:06.804 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:11:06.804 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T14:11:06.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509446 2024-03-09T14:11:06.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509446 2024-03-09T14:11:06.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836502 1-42949672969 2-64424509446' 2024-03-09T14:11:06.907 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:11:06.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836502 2024-03-09T14:11:06.908 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:11:06.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:11:06.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836502 2024-03-09T14:11:06.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:11:06.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836502 2024-03-09T14:11:06.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-03-09T14:11:06.911 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 21474836502 2024-03-09T14:11:06.912 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:11:07.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836501 -lt 21474836502 2024-03-09T14:11:07.184 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:11:08.185 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:11:08.185 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:11:08.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836502 -lt 21474836502 2024-03-09T14:11:08.469 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:11:08.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672969 2024-03-09T14:11:08.470 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:11:08.471 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:11:08.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672969 2024-03-09T14:11:08.472 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:11:08.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672969 2024-03-09T14:11:08.473 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-03-09T14:11:08.473 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 42949672969 2024-03-09T14:11:08.473 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:11:08.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672969 -lt 42949672969 2024-03-09T14:11:08.747 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:11:08.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509446 2024-03-09T14:11:08.747 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:11:08.748 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T14:11:08.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509446 2024-03-09T14:11:08.749 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:11:08.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509446 2024-03-09T14:11:08.750 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-03-09T14:11:08.751 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509446 2024-03-09T14:11:08.751 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T14:11:09.024 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509446 -lt 64424509446 2024-03-09T14:11:09.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:11:09.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:11:09.025 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:11:09.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T14:11:09.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:11:09.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:11:09.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:11:09.364 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:11:09.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:11:09.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:11:09.365 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:11:09.635 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-03-09T14:11:09.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:11:09.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:11:09.637 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:11:09.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-03-09T14:11:09.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-03-09T14:11:09.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-03-09T14:11:09.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-03-09T14:11:09.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-03-09T14:11:09.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-03-09T14:11:10.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-03-09T14:11:10.075 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:11:10.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:11:10.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:11:10.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:11:10.075 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:11:10.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:11:10.076 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:11:10.344 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T14:11:10.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:11:10.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:11:10.345 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:11:10.707 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T14:11:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:11:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:11:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: create_ec_pool: return 0 2024-03-09T14:11:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats_ec: local payload=ABCDEF 2024-03-09T14:11:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:746: TEST_repair_stats_ec: echo ABCDEF 2024-03-09T14:11:10.708 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: seq 1 30 2024-03-09T14:11:10.708 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:10.709 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:10.854 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:10.855 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:11.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:11.073 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:11.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:11.299 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:11.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:11.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:11.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:11.734 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:11.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:11.969 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:12.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:12.162 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:12.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:12.437 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:12.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:12.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:13.101 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:13.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:13.277 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:13.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:13.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:13.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:13.772 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:13.973 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:13.974 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:14.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:14.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:14.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:14.426 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:14.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:14.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:14.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:14.922 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:15.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:15.140 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:15.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:15.366 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:15.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:15.576 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:15.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:15.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:16.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:16.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:16.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:16.272 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:16.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:16.465 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:16.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:16.700 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:16.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:16.935 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:17.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:17.136 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:17.362 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2024-03-09T14:11:17.363 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2024-03-09T14:11:17.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: get_not_primary testpool obj1 2024-03-09T14:11:17.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=testpool 2024-03-09T14:11:17.648 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-03-09T14:11:17.649 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary testpool obj1 2024-03-09T14:11:17.649 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T14:11:17.649 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T14:11:17.649 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T14:11:17.649 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T14:11:17.933 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-03-09T14:11:17.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map testpool obj1 2024-03-09T14:11:17.934 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-03-09T14:11:18.208 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:754: TEST_repair_stats_ec: local other=0 2024-03-09T14:11:18.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: get_pg testpool obj1 2024-03-09T14:11:18.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=testpool 2024-03-09T14:11:18.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-03-09T14:11:18.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map testpool obj1 2024-03-09T14:11:18.209 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-03-09T14:11:18.501 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats_ec: local pgid=1.0 2024-03-09T14:11:18.502 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: get_primary testpool obj1 2024-03-09T14:11:18.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=testpool 2024-03-09T14:11:18.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-03-09T14:11:18.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map testpool obj1 2024-03-09T14:11:18.503 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-03-09T14:11:18.780 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats_ec: local primary=1 2024-03-09T14:11:18.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:758: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2024-03-09T14:11:18.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:11:18.781 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:11:18.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:11:18.781 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:11:18.782 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:11:19.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:11:19.087 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2024-03-09T14:11:19.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:11:19.088 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:11:19.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:11:19.088 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:11:19.089 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:11:19.395 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:11:19.396 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: seq 1 26 2024-03-09T14:11:19.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:19.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 1 % 2 2024-03-09T14:11:19.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T14:11:19.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2024-03-09T14:11:19.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:19.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:19.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:11:19.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:19.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:19.399 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2024-03-09T14:11:20.742 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#1:ff7b1f36:::obj1:head# 2024-03-09T14:11:21.260 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:21.260 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 2 % 2 2024-03-09T14:11:21.262 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T14:11:21.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2024-03-09T14:11:21.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:21.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:21.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:11:21.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:21.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:21.263 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2024-03-09T14:11:22.340 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#1:104778fc:::obj2:head# 2024-03-09T14:11:22.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:22.860 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 3 % 2 2024-03-09T14:11:22.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T14:11:22.862 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2024-03-09T14:11:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:11:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:22.863 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2024-03-09T14:11:23.608 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#1:8dd16f86:::obj3:head# 2024-03-09T14:11:24.126 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:24.126 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 4 % 2 2024-03-09T14:11:24.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T14:11:24.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2024-03-09T14:11:24.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:24.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:24.128 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:11:24.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:24.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:24.129 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2024-03-09T14:11:24.969 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#1:0ee9ae15:::obj4:head# 2024-03-09T14:11:25.484 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:25.484 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 5 % 2 2024-03-09T14:11:25.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T14:11:25.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2024-03-09T14:11:25.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:25.486 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:25.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:11:25.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:25.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:25.487 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2024-03-09T14:11:26.146 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#1:c52c9666:::obj5:head# 2024-03-09T14:11:26.661 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:26.662 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 6 % 2 2024-03-09T14:11:26.663 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T14:11:26.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2024-03-09T14:11:26.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:26.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:26.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:11:26.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:26.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:26.664 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2024-03-09T14:11:27.315 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#1:233a42c1:::obj6:head# 2024-03-09T14:11:27.830 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:27.830 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 7 % 2 2024-03-09T14:11:27.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T14:11:27.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2024-03-09T14:11:27.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:27.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:27.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:11:27.832 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:27.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:27.833 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2024-03-09T14:11:28.651 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#1:9a09113a:::obj7:head# 2024-03-09T14:11:29.170 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:29.170 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 8 % 2 2024-03-09T14:11:29.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T14:11:29.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2024-03-09T14:11:29.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:29.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:29.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:11:29.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:29.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:29.173 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2024-03-09T14:11:29.837 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#1:a557efb1:::obj8:head# 2024-03-09T14:11:30.351 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:30.352 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 9 % 2 2024-03-09T14:11:30.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T14:11:30.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2024-03-09T14:11:30.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:30.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:30.354 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:11:30.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:30.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:30.355 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2024-03-09T14:11:31.031 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#1:fe351e27:::obj9:head# 2024-03-09T14:11:31.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:31.553 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 10 % 2 2024-03-09T14:11:31.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T14:11:31.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2024-03-09T14:11:31.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:31.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:31.555 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:11:31.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:31.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:31.556 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2024-03-09T14:11:32.199 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#1:d1337354:::obj10:head# 2024-03-09T14:11:32.714 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:32.714 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 11 % 2 2024-03-09T14:11:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T14:11:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2024-03-09T14:11:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:11:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:32.716 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2024-03-09T14:11:33.352 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#1:eeae2a94:::obj11:head# 2024-03-09T14:11:34.117 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:34.118 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 12 % 2 2024-03-09T14:11:34.119 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T14:11:34.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2024-03-09T14:11:34.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:34.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:34.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:11:34.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:34.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:34.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2024-03-09T14:11:34.864 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#1:af5f95cb:::obj12:head# 2024-03-09T14:11:35.631 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:35.631 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 13 % 2 2024-03-09T14:11:35.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T14:11:35.633 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2024-03-09T14:11:35.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:35.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:35.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:11:35.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:35.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:35.634 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2024-03-09T14:11:36.250 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#1:a61ad63e:::obj13:head# 2024-03-09T14:11:36.765 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:36.765 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 14 % 2 2024-03-09T14:11:36.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T14:11:36.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2024-03-09T14:11:36.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:36.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:36.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:11:36.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:36.767 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:36.768 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2024-03-09T14:11:37.394 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#1:e0a44829:::obj14:head# 2024-03-09T14:11:37.909 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:37.909 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 15 % 2 2024-03-09T14:11:37.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T14:11:37.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2024-03-09T14:11:37.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:37.911 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:37.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:11:37.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:37.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:37.912 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2024-03-09T14:11:38.696 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#1:ab946124:::obj15:head# 2024-03-09T14:11:39.211 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:39.211 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 16 % 2 2024-03-09T14:11:39.213 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T14:11:39.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2024-03-09T14:11:39.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:39.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:39.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:11:39.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:39.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:39.214 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2024-03-09T14:11:39.864 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#1:461f8b5e:::obj16:head# 2024-03-09T14:11:40.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:40.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 17 % 2 2024-03-09T14:11:40.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T14:11:40.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2024-03-09T14:11:40.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:40.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:40.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:11:40.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:40.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:40.384 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2024-03-09T14:11:41.017 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#1:24fc6e92:::obj17:head# 2024-03-09T14:11:41.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:41.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 18 % 2 2024-03-09T14:11:41.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T14:11:41.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2024-03-09T14:11:41.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:41.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:41.536 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:11:41.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:41.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:41.537 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2024-03-09T14:11:42.176 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#1:b9836a99:::obj18:head# 2024-03-09T14:11:42.694 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:42.696 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 19 % 2 2024-03-09T14:11:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T14:11:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2024-03-09T14:11:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:11:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:42.696 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2024-03-09T14:11:43.504 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#1:00b31b6c:::obj19:head# 2024-03-09T14:11:44.023 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:44.023 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 20 % 2 2024-03-09T14:11:44.025 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T14:11:44.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2024-03-09T14:11:44.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:44.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:44.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:11:44.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:44.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:44.026 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2024-03-09T14:11:44.790 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#1:24b5611b:::obj20:head# 2024-03-09T14:11:45.304 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:45.305 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 21 % 2 2024-03-09T14:11:45.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T14:11:45.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2024-03-09T14:11:45.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:45.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:45.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:11:45.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:45.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:45.307 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2024-03-09T14:11:45.967 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#1:1243fe1a:::obj21:head# 2024-03-09T14:11:46.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:46.489 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 22 % 2 2024-03-09T14:11:46.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T14:11:46.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2024-03-09T14:11:46.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:46.491 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:46.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:11:46.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:46.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:46.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2024-03-09T14:11:47.202 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#1:c0c49def:::obj22:head# 2024-03-09T14:11:47.720 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:47.721 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 23 % 2 2024-03-09T14:11:47.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T14:11:47.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2024-03-09T14:11:47.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:47.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:47.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:11:47.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:47.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:47.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2024-03-09T14:11:48.487 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#1:242c40e0:::obj23:head# 2024-03-09T14:11:49.002 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:49.002 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 24 % 2 2024-03-09T14:11:49.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T14:11:49.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2024-03-09T14:11:49.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:49.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:49.004 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:11:49.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:49.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:49.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2024-03-09T14:11:49.657 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#1:7eacede5:::obj24:head# 2024-03-09T14:11:50.172 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:50.172 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 25 % 2 2024-03-09T14:11:50.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=1 2024-03-09T14:11:50.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2024-03-09T14:11:50.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:50.174 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:50.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-03-09T14:11:50.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:50.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:50.175 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2024-03-09T14:11:50.868 INFO:tasks.workunit.client.0.smithi162.stdout:remove 0#1:7109478f:::obj25:head# 2024-03-09T14:11:51.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2024-03-09T14:11:51.383 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: expr 26 % 2 2024-03-09T14:11:51.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:763: TEST_repair_stats_ec: OSD=0 2024-03-09T14:11:51.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:764: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2024-03-09T14:11:51.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2024-03-09T14:11:51.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-03-09T14:11:51.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-03-09T14:11:51.385 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-03-09T14:11:51.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:51.386 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2024-03-09T14:11:52.053 INFO:tasks.workunit.client.0.smithi162.stdout:remove 1#1:f3c2593a:::obj26:head# 2024-03-09T14:11:52.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:11:52.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T14:11:52.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T14:11:52.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-03-09T14:11:52.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T14:11:52.571 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/1 2024-03-09T14:11:52.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:11:52.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:11:52.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:11:52.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:11:52.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2024-03-09T14:11:52.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2024-03-09T14:11:52.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T14:11:52.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T14:11:52.572 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:11:52.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T14:11:52.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:11:52.572 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:11:52.572 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T14:11:52.573 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/1 2024-03-09T14:11:52.574 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-03-09T14:11:52.574 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.1 2024-03-09T14:11:52.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:11:52.575 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/1/whoami 2024-03-09T14:11:52.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-03-09T14:11:52.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T14:11:52.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T14:11:52.575 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T14:11:52.603 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:11:52.602+0000 7f4f3680f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:11:52.605 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:11:52.605+0000 7f4f3680f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:11:52.606 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:11:52.606+0000 7f4f3680f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:11:52.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-03-09T14:11:52.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:11:52.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-03-09T14:11:52.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:11:52.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:11:52.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:11:52.859 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:11:52.859 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:11:52.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:11:52.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:11:53.132 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:11:53.133 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:11:53.133+0000 7f4f3680f640 -1 Falling back to public interface 2024-03-09T14:11:54.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:11:54.134 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:11:54.134 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:11:54.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:11:54.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:11:54.135 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:11:54.420 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:11:55.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:11:55.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:11:55.421 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:11:55.421 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:11:55.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:11:55.422 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:11:55.423 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:11:55.423+0000 7f4f3680f640 -1 osd.1 22 log_to_monitors true 2024-03-09T14:11:55.723 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:11:56.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:11:56.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:11:56.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:11:56.725 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:11:56.725 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:11:56.726 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-03-09T14:11:57.002 INFO:tasks.workunit.client.0.smithi162.stdout:osd.1 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [10,22) [v2:127.0.0.1:6802/3128183020,v1:127.0.0.1:6803/3128183020] [v2:127.0.0.1:6804/3128183020,v1:127.0.0.1:6805/3128183020] exists,up e6322f9f-8376-4351-962b-cb56b8af7f58 2024-03-09T14:11:57.005 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:11:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:11:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:11:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:11:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-scrub-repair 2024-03-09T14:11:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-03-09T14:11:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-03-09T14:11:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-03-09T14:11:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-scrub-repair/0 2024-03-09T14:11:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2024-03-09T14:11:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-03-09T14:11:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-03-09T14:11:57.006 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-03-09T14:11:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2024-03-09T14:11:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2024-03-09T14:11:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-03-09T14:11:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-03-09T14:11:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2024-03-09T14:11:57.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-03-09T14:11:57.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-03-09T14:11:57.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-03-09T14:11:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-03-09T14:11:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:11:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:11:57.007 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:11:57.007 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' 2024-03-09T14:11:57.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-03-09T14:11:57.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2024-03-09T14:11:57.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2024-03-09T14:11:57.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-03-09T14:11:57.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-03-09T14:11:57.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-03-09T14:11:57.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-03-09T14:11:57.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-03-09T14:11:57.008 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2024-03-09T14:11:57.009 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-scrub-repair/0 2024-03-09T14:11:57.009 INFO:tasks.workunit.client.0.smithi162.stdout:start osd.0 2024-03-09T14:11:57.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-03-09T14:11:57.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52fb397d-19b9-4411-b4dc-d406f734e662 --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.70042/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2024-03-09T14:11:57.010 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-scrub-repair/0/whoami 2024-03-09T14:11:57.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-03-09T14:11:57.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-03-09T14:11:57.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-03-09T14:11:57.010 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-03-09T14:11:57.031 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:11:57.031+0000 7f5d2780f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:11:57.033 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:11:57.033+0000 7f5d2780f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:11:57.034 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:11:57.034+0000 7f5d2780f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-03-09T14:11:57.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-03-09T14:11:57.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-03-09T14:11:57.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-03-09T14:11:57.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-03-09T14:11:57.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-03-09T14:11:57.279 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:11:57.280 INFO:tasks.workunit.client.0.smithi162.stdout:0 2024-03-09T14:11:57.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-03-09T14:11:57.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:11:57.280 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:11:57.551 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:11:57.555 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:11:57.555+0000 7f5d2780f640 -1 Falling back to public interface 2024-03-09T14:11:58.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:11:58.552 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:11:58.553 INFO:tasks.workunit.client.0.smithi162.stdout:1 2024-03-09T14:11:58.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-03-09T14:11:58.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:11:58.553 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:11:58.834 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:11:59.835 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:11:59.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:11:59.836 INFO:tasks.workunit.client.0.smithi162.stdout:2 2024-03-09T14:11:59.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-03-09T14:11:59.836 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:11:59.837 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:11:59.870 INFO:tasks.workunit.client.0.smithi162.stderr:2024-03-09T14:11:59.869+0000 7f5d2780f640 -1 osd.0 21 log_to_monitors true 2024-03-09T14:12:00.158 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-03-09T14:12:01.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-03-09T14:12:01.159 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-03-09T14:12:01.159 INFO:tasks.workunit.client.0.smithi162.stdout:3 2024-03-09T14:12:01.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-03-09T14:12:01.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-03-09T14:12:01.160 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-03-09T14:12:01.440 INFO:tasks.workunit.client.0.smithi162.stdout:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6810/1062881204,v1:127.0.0.1:6811/1062881204] [v2:127.0.0.1:6812/1062881204,v1:127.0.0.1:6813/1062881204] exists,up 66d1bcfe-667e-4ef5-a21b-5224e96e1898 2024-03-09T14:12:01.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-03-09T14:12:01.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-03-09T14:12:01.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-03-09T14:12:01.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:768: TEST_repair_stats_ec: wait_for_clean 2024-03-09T14:12:01.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:12:01.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:12:01.442 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:12:01.442 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:12:01.442 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:12:01.442 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:12:01.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:12:01.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:12:01.443 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:12:01.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:12:01.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:12:01.533 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:12:01.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:12:01.534 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:12:01.534 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:12:01.806 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:12:01.806 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:12:01.807 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T14:12:01.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:12:01.807 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:12:01.807 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:12:01.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051586 2024-03-09T14:12:01.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051586 2024-03-09T14:12:01.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586' 2024-03-09T14:12:01.910 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:12:01.910 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:12:02.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149699 2024-03-09T14:12:02.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149699 2024-03-09T14:12:02.014 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-111669149699' 2024-03-09T14:12:02.015 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:12:02.015 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T14:12:02.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509458 2024-03-09T14:12:02.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509458 2024-03-09T14:12:02.120 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051586 1-111669149699 2-64424509458' 2024-03-09T14:12:02.121 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:12:02.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051586 2024-03-09T14:12:02.121 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:12:02.122 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:12:02.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051586 2024-03-09T14:12:02.123 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:12:02.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051586 2024-03-09T14:12:02.124 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2024-03-09T14:12:02.124 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 124554051586 2024-03-09T14:12:02.125 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:12:02.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 124554051586 2024-03-09T14:12:02.396 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:12:03.397 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:12:03.397 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:12:03.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051586 -lt 124554051586 2024-03-09T14:12:03.682 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:12:03.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149699 2024-03-09T14:12:03.683 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:12:03.684 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:12:03.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149699 2024-03-09T14:12:03.684 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:12:03.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149699 2024-03-09T14:12:03.686 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149699' 2024-03-09T14:12:03.686 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 111669149699 2024-03-09T14:12:03.686 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:12:03.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149699 -lt 111669149699 2024-03-09T14:12:03.960 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:12:03.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509458 2024-03-09T14:12:03.961 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:12:03.962 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T14:12:03.962 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509458 2024-03-09T14:12:03.963 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:12:03.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509458 2024-03-09T14:12:03.964 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509458 2024-03-09T14:12:03.964 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-03-09T14:12:03.964 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T14:12:04.236 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509458 -lt 64424509458 2024-03-09T14:12:04.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:12:04.237 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:12:04.238 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:12:04.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T14:12:04.582 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:12:04.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:12:04.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:12:04.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:12:04.583 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:12:04.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:12:04.584 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:12:04.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T14:12:04.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:12:04.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:12:04.870 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:12:05.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T14:12:05.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:12:05.216 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:12:05.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats_ec: repair 1.0 2024-03-09T14:12:05.217 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-03-09T14:12:05.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-03-09T14:12:05.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T14:12:05.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:12:05.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:12:05.217 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T14:12:05.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-03-09T14:11:04.993562+0000 2024-03-09T14:12:05.489 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-03-09T14:12:05.745 INFO:tasks.workunit.client.0.smithi162.stderr:instructing pg 1.0s0 on osd.1 to repair 2024-03-09T14:12:05.758 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-03-09T14:11:04.993562+0000 2024-03-09T14:12:05.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: wait_for_scrub: local pgid=1.0 2024-03-09T14:12:05.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: wait_for_scrub: local last_scrub=2024-03-09T14:11:04.993562+0000 2024-03-09T14:12:05.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2011: wait_for_scrub: local sname=last_scrub_stamp 2024-03-09T14:12:05.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i=0 )) 2024-03-09T14:12:05.759 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:12:05.759 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T14:12:05.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T14:12:05.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:12:05.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:12:05.760 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T14:12:06.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:11:04.993562+0000 '>' 2024-03-09T14:11:04.993562+0000 2024-03-09T14:12:06.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:12:07.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:12:07.027 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:12:07.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T14:12:07.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T14:12:07.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:12:07.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:12:07.028 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T14:12:07.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:11:04.993562+0000 '>' 2024-03-09T14:11:04.993562+0000 2024-03-09T14:12:07.305 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:12:08.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:12:08.306 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:12:08.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T14:12:08.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T14:12:08.307 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:12:08.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:12:08.308 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T14:12:08.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:11:04.993562+0000 '>' 2024-03-09T14:11:04.993562+0000 2024-03-09T14:12:08.588 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:12:09.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:12:09.589 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:12:09.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T14:12:09.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T14:12:09.589 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:12:09.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:12:09.590 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T14:12:09.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:11:04.993562+0000 '>' 2024-03-09T14:11:04.993562+0000 2024-03-09T14:12:09.869 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:12:10.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:12:10.870 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:12:10.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T14:12:10.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T14:12:10.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:12:10.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:12:10.871 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T14:12:11.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:11:04.993562+0000 '>' 2024-03-09T14:11:04.993562+0000 2024-03-09T14:12:11.145 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: wait_for_scrub: sleep 1 2024-03-09T14:12:12.146 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i++ )) 2024-03-09T14:12:12.147 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2013: wait_for_scrub: (( i < 300 )) 2024-03-09T14:12:12.147 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-03-09T14:12:12.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-03-09T14:12:12.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-03-09T14:12:12.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-03-09T14:12:12.148 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-03-09T14:12:12.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2014: wait_for_scrub: test 2024-03-09T14:12:06.576878+0000 '>' 2024-03-09T14:11:04.993562+0000 2024-03-09T14:12:12.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: wait_for_scrub: return 0 2024-03-09T14:12:12.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats_ec: wait_for_clean 2024-03-09T14:12:12.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-03-09T14:12:12.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-03-09T14:12:12.429 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-03-09T14:12:12.429 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-03-09T14:12:12.430 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-03-09T14:12:12.430 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-03-09T14:12:12.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-03-09T14:12:12.430 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-03-09T14:12:12.431 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-03-09T14:12:12.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-03-09T14:12:12.521 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-03-09T14:12:12.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-03-09T14:12:12.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-03-09T14:12:12.522 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:12:12.522 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:12:12.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:12:12.790 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:12:12.790 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T14:12:12.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:12:12.790 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:12:12.790 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:12:12.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051589 2024-03-09T14:12:12.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051589 2024-03-09T14:12:12.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589' 2024-03-09T14:12:12.896 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:12:12.896 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:12:12.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149702 2024-03-09T14:12:12.999 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149702 2024-03-09T14:12:13.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-111669149702' 2024-03-09T14:12:13.000 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:12:13.000 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T14:12:13.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509461 2024-03-09T14:12:13.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509461 2024-03-09T14:12:13.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051589 1-111669149702 2-64424509461' 2024-03-09T14:12:13.105 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:12:13.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051589 2024-03-09T14:12:13.106 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:12:13.107 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:12:13.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051589 2024-03-09T14:12:13.108 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:12:13.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051589 2024-03-09T14:12:13.109 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051589' 2024-03-09T14:12:13.109 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 124554051589 2024-03-09T14:12:13.109 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:12:13.380 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051588 -lt 124554051589 2024-03-09T14:12:13.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:12:14.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:12:14.382 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:12:14.665 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051589 -lt 124554051589 2024-03-09T14:12:14.666 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:12:14.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149702 2024-03-09T14:12:14.666 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:12:14.667 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:12:14.668 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149702 2024-03-09T14:12:14.668 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:12:14.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149702 2024-03-09T14:12:14.670 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149702' 2024-03-09T14:12:14.670 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 111669149702 2024-03-09T14:12:14.670 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:12:14.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149702 -lt 111669149702 2024-03-09T14:12:14.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:12:14.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509461 2024-03-09T14:12:14.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:12:14.946 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T14:12:14.946 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509461 2024-03-09T14:12:14.947 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:12:14.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509461 2024-03-09T14:12:14.948 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509461 2024-03-09T14:12:14.948 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-03-09T14:12:14.948 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T14:12:15.220 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509462 -lt 64424509461 2024-03-09T14:12:15.220 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-03-09T14:12:15.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:12:15.221 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:12:15.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-03-09T14:12:15.591 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-03-09T14:12:15.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-03-09T14:12:15.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-03-09T14:12:15.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-03-09T14:12:15.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-03-09T14:12:15.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-03-09T14:12:15.592 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-03-09T14:12:15.860 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-03-09T14:12:15.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-03-09T14:12:15.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-03-09T14:12:15.861 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-03-09T14:12:16.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-03-09T14:12:16.198 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-03-09T14:12:16.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-03-09T14:12:16.199 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:772: TEST_repair_stats_ec: ceph pg dump pgs 2024-03-09T14:12:16.455 INFO:tasks.workunit.client.0.smithi162.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-03-09T14:12:16.491 INFO:tasks.workunit.client.0.smithi162.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2024-03-09T14:12:07.817458+0000 21'30 30:130 [1,0,2] 1 [1,0,2] 1 21'30 2024-03-09T14:12:06.576878+0000 21'30 2024-03-09T14:12:06.576878+0000 0 1 periodic scrub scheduled @ 2024-03-10T14:12:06.576878+0000 17 0 2024-03-09T14:12:16.491 INFO:tasks.workunit.client.0.smithi162.stdout: 2024-03-09T14:12:16.491 INFO:tasks.workunit.client.0.smithi162.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-03-09T14:12:16.492 INFO:tasks.workunit.client.0.smithi162.stderr:dumped pgs 2024-03-09T14:12:16.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats_ec: flush_pg_stats 2024-03-09T14:12:16.492 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-03-09T14:12:16.492 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-03-09T14:12:16.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-03-09T14:12:16.737 INFO:tasks.workunit.client.0.smithi162.stderr:1 2024-03-09T14:12:16.737 INFO:tasks.workunit.client.0.smithi162.stderr:2' 2024-03-09T14:12:16.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-03-09T14:12:16.737 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:12:16.737 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-03-09T14:12:16.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=124554051591 2024-03-09T14:12:16.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 124554051591 2024-03-09T14:12:16.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591' 2024-03-09T14:12:16.840 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:12:16.840 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-03-09T14:12:16.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=111669149704 2024-03-09T14:12:16.944 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 111669149704 2024-03-09T14:12:16.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-111669149704' 2024-03-09T14:12:16.945 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-03-09T14:12:16.945 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-03-09T14:12:17.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509463 2024-03-09T14:12:17.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509463 2024-03-09T14:12:17.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-124554051591 1-111669149704 2-64424509463' 2024-03-09T14:12:17.047 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:12:17.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-124554051591 2024-03-09T14:12:17.047 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:12:17.048 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-03-09T14:12:17.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-124554051591 2024-03-09T14:12:17.049 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:12:17.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=124554051591 2024-03-09T14:12:17.050 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.0 seq 124554051591 2024-03-09T14:12:17.051 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2024-03-09T14:12:17.051 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:12:17.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051590 -lt 124554051591 2024-03-09T14:12:17.321 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-03-09T14:12:18.322 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-03-09T14:12:18.323 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-03-09T14:12:18.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 124554051591 -lt 124554051591 2024-03-09T14:12:18.609 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:12:18.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-111669149704 2024-03-09T14:12:18.610 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:12:18.611 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-03-09T14:12:18.611 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-111669149704 2024-03-09T14:12:18.612 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:12:18.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=111669149704 2024-03-09T14:12:18.613 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 111669149704' 2024-03-09T14:12:18.613 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.1 seq 111669149704 2024-03-09T14:12:18.613 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-03-09T14:12:18.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 111669149704 -lt 111669149704 2024-03-09T14:12:18.888 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-03-09T14:12:18.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509463 2024-03-09T14:12:18.889 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-03-09T14:12:18.889 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-03-09T14:12:18.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509463 2024-03-09T14:12:18.890 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-03-09T14:12:18.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509463 2024-03-09T14:12:18.892 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2024-03-09T14:12:18.892 INFO:tasks.workunit.client.0.smithi162.stdout:waiting osd.2 seq 64424509463 2024-03-09T14:12:18.892 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-03-09T14:12:19.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509463 -lt 64424509463 2024-03-09T14:12:19.164 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: ceph pg 1.0 query 2024-03-09T14:12:19.165 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:776: TEST_repair_stats_ec: jq .info.stats.stat_sum 2024-03-09T14:12:19.271 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T14:12:19.271 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes": 210, 2024-03-09T14:12:19.271 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects": 30, 2024-03-09T14:12:19.271 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_clones": 0, 2024-03-09T14:12:19.271 INFO:tasks.workunit.client.0.smithi162.stdout: "num_object_copies": 90, 2024-03-09T14:12:19.271 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing_on_primary": 0, 2024-03-09T14:12:19.271 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_missing": 0, 2024-03-09T14:12:19.271 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_degraded": 0, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_misplaced": 0, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_unfound": 0, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_dirty": 30, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_whiteouts": 0, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read": 0, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_read_kb": 0, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write": 30, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_write_kb": 30, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_scrub_errors": 0, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shallow_scrub_errors": 0, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_deep_scrub_errors": 0, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_recovered": 26, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_recovered": 182, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_keys_recovered": 0, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_omap": 0, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_hit_set_archive": 0, 2024-03-09T14:12:19.272 INFO:tasks.workunit.client.0.smithi162.stdout: "num_bytes_hit_set_archive": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_kb": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_kb": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_promote": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_high": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_flush_mode_low": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_some": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_evict_mode_full": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_pinned": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_legacy_snapsets": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_large_omap_objects": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_manifest": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_bytes": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_omap_keys": 0, 2024-03-09T14:12:19.273 INFO:tasks.workunit.client.0.smithi162.stdout: "num_objects_repaired": 26 2024-03-09T14:12:19.274 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T14:12:19.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: ceph pg 1.0 query 2024-03-09T14:12:19.274 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2024-03-09T14:12:19.378 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats_ec: COUNT=26 2024-03-09T14:12:19.379 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats_ec: test 26 = 26 2024-03-09T14:12:19.379 INFO:tasks.workunit.client.0.smithi162.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: expr 3 - 1 2024-03-09T14:12:19.380 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: seq 0 2 2024-03-09T14:12:19.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:780: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-03-09T14:12:19.381 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:781: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2024-03-09T14:12:19.381 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: expr 26 / 2 2024-03-09T14:12:19.382 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:782: TEST_repair_stats_ec: repair=13 2024-03-09T14:12:19.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-03-09T14:12:19.383 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:787: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2024-03-09T14:12:19.639 INFO:tasks.workunit.client.0.smithi162.stderr:dumped all 2024-03-09T14:12:19.650 INFO:tasks.workunit.client.0.smithi162.stdout:{ 2024-03-09T14:12:19.650 INFO:tasks.workunit.client.0.smithi162.stdout: "osd": 0, 2024-03-09T14:12:19.651 INFO:tasks.workunit.client.0.smithi162.stdout: "up_from": 29, 2024-03-09T14:12:19.651 INFO:tasks.workunit.client.0.smithi162.stdout: "seq": 124554051591, 2024-03-09T14:12:19.651 INFO:tasks.workunit.client.0.smithi162.stdout: "num_pgs": 1, 2024-03-09T14:12:19.651 INFO:tasks.workunit.client.0.smithi162.stdout: "num_osds": 1, 2024-03-09T14:12:19.651 INFO:tasks.workunit.client.0.smithi162.stdout: "num_per_pool_osds": 1, 2024-03-09T14:12:19.651 INFO:tasks.workunit.client.0.smithi162.stdout: "num_per_pool_omap_osds": 1, 2024-03-09T14:12:19.651 INFO:tasks.workunit.client.0.smithi162.stdout: "kb": 104857600, 2024-03-09T14:12:19.651 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used": 28472, 2024-03-09T14:12:19.651 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used_data": 368, 2024-03-09T14:12:19.651 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used_omap": 4, 2024-03-09T14:12:19.652 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_used_meta": 28091, 2024-03-09T14:12:19.652 INFO:tasks.workunit.client.0.smithi162.stdout: "kb_avail": 104829128, 2024-03-09T14:12:19.652 INFO:tasks.workunit.client.0.smithi162.stdout: "statfs": { 2024-03-09T14:12:19.652 INFO:tasks.workunit.client.0.smithi162.stdout: "total": 107374182400, 2024-03-09T14:12:19.652 INFO:tasks.workunit.client.0.smithi162.stdout: "available": 107345027072, 2024-03-09T14:12:19.652 INFO:tasks.workunit.client.0.smithi162.stdout: "internally_reserved": 0, 2024-03-09T14:12:19.652 INFO:tasks.workunit.client.0.smithi162.stdout: "allocated": 376832, 2024-03-09T14:12:19.652 INFO:tasks.workunit.client.0.smithi162.stdout: "data_stored": 202971, 2024-03-09T14:12:19.652 INFO:tasks.workunit.client.0.smithi162.stdout: "data_compressed": 0, 2024-03-09T14:12:19.652 INFO:tasks.workunit.client.0.smithi162.stdout: "data_compressed_allocated": 0, 2024-03-09T14:12:19.652 INFO:tasks.workunit.client.0.smithi162.stdout: "data_compressed_original": 0, 2024-03-09T14:12:19.652 INFO:tasks.workunit.client.0.smithi162.stdout: "omap_allocated": 4402, 2024-03-09T14:12:19.653 INFO:tasks.workunit.client.0.smithi162.stdout: "internal_metadata": 28765902 2024-03-09T14:12:19.653 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T14:12:19.653 INFO:tasks.workunit.client.0.smithi162.stdout: "hb_peers": [ 2024-03-09T14:12:19.653 INFO:tasks.workunit.client.0.smithi162.stdout: 1, 2024-03-09T14:12:19.653 INFO:tasks.workunit.client.0.smithi162.stdout: 2 2024-03-09T14:12:19.653 INFO:tasks.workunit.client.0.smithi162.stdout: ], 2024-03-09T14:12:19.653 INFO:tasks.workunit.client.0.smithi162.stdout: "snap_trim_queue_len": 0, 2024-03-09T14:12:19.653 INFO:tasks.workunit.client.0.smithi162.stdout: "num_snap_trimming": 0, 2024-03-09T14:12:19.653 INFO:tasks.workunit.client.0.smithi162.stdout: "num_shards_repaired": 26, 2024-03-09T14:12:19.653 INFO:tasks.workunit.client.0.smithi162.stdout: "op_queue_age_hist": { 2024-03-09T14:12:19.653 INFO:tasks.workunit.client.0.smithi162.stdout: "histogram": [], 2024-03-09T14:12:19.653 INFO:tasks.workunit.client.0.smithi162.stdout: "upper_bound": 1 2024-03-09T14:12:19.654 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T14:12:19.654 INFO:tasks.workunit.client.0.smithi162.stdout: "perf_stat": { 2024-03-09T14:12:19.654 INFO:tasks.workunit.client.0.smithi162.stdout: "commit_latency_ms": 0, 2024-03-09T14:12:19.654 INFO:tasks.workunit.client.0.smithi162.stdout: "apply_latency_ms": 0, 2024-03-09T14:12:19.654 INFO:tasks.workunit.client.0.smithi162.stdout: "commit_latency_ns": 0, 2024-03-09T14:12:19.654 INFO:tasks.workunit.client.0.smithi162.stdout: "apply_latency_ns": 0 2024-03-09T14:12:19.654 INFO:tasks.workunit.client.0.smithi162.stdout: }, 2024-03-09T14:12:19.654 INFO:tasks.workunit.client.0.smithi162.stdout: "alerts": [] 2024-03-09T14:12:19.654 INFO:tasks.workunit.client.0.smithi162.stdout:} 2024-03-09T14:12:19.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2024-03-09T14:12:19.655 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2024-03-09T14:12:19.911 INFO:tasks.workunit.client.0.smithi162.stderr:dumped all 2024-03-09T14:12:19.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: COUNT=26 2024-03-09T14:12:19.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: test 26 = 13 2024-03-09T14:12:19.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: return 1 2024-03-09T14:12:19.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: return 1 2024-03-09T14:12:19.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: code=1 2024-03-09T14:12:19.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/osd-scrub-repair 1 2024-03-09T14:12:19.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-scrub-repair 2024-03-09T14:12:19.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-03-09T14:12:19.924 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-scrub-repair KILL 2024-03-09T14:12:19.924 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-03-09T14:12:19.925 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-03-09T14:12:19.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-03-09T14:12:19.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-03-09T14:12:19.925 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-03-09T14:12:20.032 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-03-09T14:12:20.033 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-03-09T14:12:20.033 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-03-09T14:12:20.034 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-03-09T14:12:20.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-03-09T14:12:20.035 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-03-09T14:12:20.035 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-03-09T14:12:20.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:12:20.036 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-03-09T14:12:20.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-03-09T14:12:20.037 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-09T14:12:20.037 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-03-09T14:12:20.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-03-09T14:12:20.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-03-09T14:12:20.039 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-03-09T14:12:20.040 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-scrub-repair/mgr.x.log td/osd-scrub-repair/mon.a.log td/osd-scrub-repair/osd.0.log td/osd-scrub-repair/osd.1.log td/osd-scrub-repair/osd.2.log /home/ubuntu/cephtest/archive/log 2024-03-09T14:12:20.041 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-scrub-repair 2024-03-09T14:12:20.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-03-09T14:12:20.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-03-09T14:12:20.050 INFO:tasks.workunit.client.0.smithi162.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.70042 2024-03-09T14:12:20.050 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.70042 2024-03-09T14:12:20.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:12:20.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-03-09T14:12:20.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-03-09T14:12:20.052 INFO:tasks.workunit.client.0.smithi162.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 1 2024-03-09T14:12:20.053 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2024-03-09T14:12:20.053 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-09T14:12:20.331 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_6b01fe95f3aa686f1951a0c5828907a52930b92a/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_6b01fe95f3aa686f1951a0c5828907a52930b92a/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_6b01fe95f3aa686f1951a0c5828907a52930b92a/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi162 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6b01fe95f3aa686f1951a0c5828907a52930b92a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-09T14:12:20.586 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f845ac637c2b487da7e3717e296b76dd Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_6b01fe95f3aa686f1951a0c5828907a52930b92a/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_6b01fe95f3aa686f1951a0c5828907a52930b92a/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_6b01fe95f3aa686f1951a0c5828907a52930b92a/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi162 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6b01fe95f3aa686f1951a0c5828907a52930b92a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 2024-03-09T14:12:20.589 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-09T14:12:20.599 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-09T14:12:20.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-09T14:12:20.651 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-09T14:12:20.660 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-09T14:12:20.660 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-09T14:12:20.684 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-03-09T14:12:20.689 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-09T14:12:20.689 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-03-09T14:12:20.689 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 54 +870us[ +870us] +/- 94ms 2024-03-09T14:12:20.690 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv02.front.sepia.ceph.com 3 7 377 112 -878us[ -850us] +/- 59ms 2024-03-09T14:12:20.690 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +626us[ +626us] +/- 75ms 2024-03-09T14:12:20.690 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-09T14:12:20.690 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-09T14:12:20.698 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-09T14:12:20.699 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-09T14:12:20.721 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-09T14:12:20.792 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-09T14:12:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:12:20.821 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-09T14:12:20.829 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-09T14:12:20.855 INFO:teuthology.task.internal:Duration was 5604.986372 seconds 2024-03-09T14:12:20.855 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-09T14:12:20.939 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-09T14:12:20.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-09T14:12:20.983 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-09T14:12:21.397 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-09T14:12:21.398 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-03-09T14:12:21.398 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-09T14:12:21.417 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-09T14:12:21.417 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-09T14:12:21.485 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-09T14:12:21.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-09T14:12:21.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-09T14:12:21.633 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-09T14:12:21.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-09T14:12:21.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-09T14:12:21.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-09T14:12:21.725 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-03-09T14:12:21.764 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-09T14:12:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:12:21.818 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-09T14:12:21.828 INFO:teuthology.task.internal:Transferring archived files... 2024-03-09T14:12:21.830 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-08_16:18:36-rados-wip-yuri8-testing-2024-03-06-1329-distro-default-smithi/7587021/remote/smithi162 2024-03-09T14:12:21.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-09T14:12:22.946 INFO:teuthology.task.internal:Removing archive directory... 2024-03-09T14:12:22.946 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-09T14:12:22.988 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-09T14:12:22.998 INFO:teuthology.task.internal:Not uploading archives. 2024-03-09T14:12:22.998 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-09T14:12:23.030 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-09T14:12:23.031 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-09T14:12:23.046 INFO:teuthology.orchestra.run.smithi162.stdout: 265312 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 9 14:12 /home/ubuntu/cephtest 2024-03-09T14:12:23.046 INFO:teuthology.orchestra.run.smithi162.stdout: 403010 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 9 12:45 /home/ubuntu/cephtest/mnt.0 2024-03-09T14:12:23.046 INFO:teuthology.orchestra.run.smithi162.stdout: 403011 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 9 12:55 /home/ubuntu/cephtest/mnt.0/client.0 2024-03-09T14:12:23.047 INFO:teuthology.orchestra.run.smithi162.stdout: 403052 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 9 12:55 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-09T14:12:23.047 INFO:teuthology.orchestra.run.smithi162.stdout: 417237 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 9 14:12 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-03-09T14:12:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T14:12:23.048 INFO:teuthology.orchestra.run.smithi162.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-09T14:12:23.048 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi162 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-09T14:12:23.053 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-09T14:12:23.064 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-09T14:12:23.098 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi162 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-09T14:12:23.098 INFO:teuthology.nuke:Checking targets against current locks 2024-03-09T14:12:23.127 DEBUG:teuthology.nuke:shortname: smithi162 2024-03-09T14:12:23.128 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T14:12:23.150 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-08_16:18:36-rados-wip-yuri8-testing-2024-03-06-1329-distro-default-smithi/7587021', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-09 12:29:47.993507', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEN5uJTU89OO65vN6U9GqKJMNEG4MsVHa5bZQ7w2+Rhh6g4fVNeoAfzK25WmUu4UiDLKUku6wEe+amWomUPmEyA='} 2024-03-09T14:12:23.201 INFO:teuthology.orchestra.console.smithi162:Power off 2024-03-09T14:12:23.202 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T14:12:23.224 DEBUG:teuthology.orchestra.console.smithi162:power off output: Chassis Power Control: Down/Off 2024-03-09T14:12:23.224 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T14:12:23.247 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-03-09T14:12:27.249 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T14:12:27.359 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is on 2024-03-09T14:12:31.361 DEBUG:teuthology.orchestra.console.smithi162:pexpect command: ipmitool -H smithi162.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T14:12:31.470 DEBUG:teuthology.orchestra.console.smithi162:check power output: Chassis Power is off 2024-03-09T14:12:31.571 INFO:teuthology.orchestra.console.smithi162:Power off completed 2024-03-09T14:12:31.677 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub} duration: 5604.986372470856 failure_reason: 'Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi162 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6b01fe95f3aa686f1951a0c5828907a52930b92a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f845ac637c2b487da7e3717e296b76dd status: fail success: false 2024-03-09T14:12:31.677 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-09T14:12:31.751 INFO:teuthology.run:FAIL